1 Reply Latest reply on Jun 20, 2016 9:40 AM by galder.zamarreno

    Replication timeout

    przemek2000

      Hi,

       

      I use 7.2 version and I have replicated cache in SYNC mode. There are two nodes which are two standalone jbosses, each is run by ./standalone.sh -c standalone-full.xml -b 172.25.122.72  -u 228.6.7.8 -Djboss.node.name=node1 (second jboss has different node name and address)

       

      I can observe in logs in both servers:

       

      Received new cluster view for channel service: [APP-21394|1] (2) [APP-21394, APP-32354]

       

      Sometimes replication works correctly but in most cases there is an error. I am sure that it is not a problem with performance. I think node1 replicates to node 2 and unfortunately node 2 back to node 1 (maybe problem is with primary owner of key after rebalance..?)

       

      Could you help mi?

       

      Thanks,

      Przemek

       

      2016-06-14 12:36:47,783 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p2-t11) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for APP-32354

        at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:87) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:586) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:339) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:105) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:150) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:96) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:388) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:453) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:164) [infinispan-query-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:88) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:40) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:55) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:317) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:249) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:107) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1619) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1099) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.cache.impl.CacheImpl.access$100(CacheImpl.java:129) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at org.infinispan.cache.impl.CacheImpl$2.call(CacheImpl.java:1233) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]

        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_40]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_40]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_40]

        at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]

        • 1. Re: Replication timeout
          galder.zamarreno

          The first thing I'd suggest is that you try a more recent version. Our current release is 8.2.2.


          Then, if problems persist I'd try to see if there's GC halting stuff in the remote node, maybe get some threads dumps to see if anything is stuck, and finally inspect TRACE enabled logs on org.infinispan package to get a deeper understanding.