org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology

[tomas11]

Hi all

With infinispan-embedded-8.2.0.Final.jar we've git following exception:

org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 14

    at org.infinispan.statetransfer.StateTransferLockImpl.reportErrorAfterWait(StateTransferLockImpl.java:159)

    at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:98)

    at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:97)

    at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:366)

    at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:281)

    at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:122)

    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

    at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)

    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

    at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

    at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)

    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:48)

    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)

    at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

    at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

    at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:48)

    at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:63)

    at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)

    at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1672)

    at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:557)

    at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:549)

    at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:543)

It is spotted when one on nodes is leaving cluster and meanwhile another node is pushing data to existing caches.

Is it something which should be covered with [JBEAP-735] Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology - JBoss Iss… and it is not?

How this can be covered?

Thanks

Tomas

[galder.zamarreno]

That jira refers specifically to running Infinispan within EAP. Are you indeed running within EAP or standalone?

[tomas11]

Hi Galder

We are running it in standalone. We can not spot this behavior or regular basis but however is unpleasant when it happens.

Can following case be reason for that: It is spotted when one on nodes is leaving cluster and meanwhile another node is pushing data to existing caches.

We have case where we have CacheManagerListener and on viewChanged we do operations where we iterate through some caches or do puts into caches. Is this fine to do or can this do bad things?

[dan.berindei]

Hi Tomas

First of all, I recommend upgrading to 8.2.1.Final, because it fixes some issues like ISPN-6399.

But your problem is most likely caused by the synchronous @ViewChanged listener. Accessing a cache in the listener is quite likely to cause a deadlock by blocking, and at the same time waiting for, our internal @ViewChanged listeners.

I recommend changing your listener to spawn an asynchronous task and do the actual work on a separate thread pool. You can also use @Listener(sync = false) to use our internal asyncThreadPool, but you'd have to change the configuration: we assume asynchronous listeners are also short, so the default asyncThreadPool only has 1 thread.