4 Replies Latest reply on Jul 19, 2013 1:19 AM by Vadivel G

    Cluster is very unstable

    Vadivel G Newbie

      Hi,

       

      I am running 2 node infinispan UDP cluster and used RadarGun to validate the performance of the data grid.

       

      The cluster is running good with smaller load. If the load is increased, cluster becoming very unstable.

      The nodes are becoming into suspect mode and disconnecting very often form the cluster which leads frequent cluster reforming.

       

      05:07:36,468 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 16729 ms, adding it to suspect list

      05:07:36,469 DEBUG [FD_ALL] suspecting [localhost-51593]

      05:07:38,472 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 18733 ms, adding it to suspect list

      05:07:38,473 DEBUG [FD_ALL] suspecting [localhost-51593]

      05:07:40,474 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 20735 ms, adding it to suspect list

      05:07:40,474 DEBUG [FD_ALL] suspecting [localhost-51593]

      05:07:41,472 DEBUG [RequestCorrelator] suspect=localhost-51593

      05:07:41,478 ERROR [InvocationContextInterceptor] ISPN000136: Execution error

      org.infinispan.remoting.transport.jgroups.SuspectException: Node localhost-51593 was suspected

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:184)

              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)

              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)

              at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)

              at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108)

              at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)

              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)

              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)

              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)

              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)

              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)

              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)

              at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)

              at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)

              at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)

              at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)

              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)

              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)

              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)

              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)

              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)

              at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1196)

              at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783)

              at org.infinispan.CacheImpl.put(CacheImpl.java:777)

              at org.infinispan.CacheImpl.put(CacheImpl.java:771)

              at org.infinispan.CacheSupport.put(CacheSupport.java:53)

              at org.radargun.cachewrappers.InfinispanWrapper.put(InfinispanWrapper.java:244)

              at org.radargun.cachewrappers.InfinispanExplicitLockingWrapper.put(InfinispanExplicitLockingWrapper.java:72)

              at org.radargun.stressors.StressTestStressor$FixedSetSharedOperationLogic.init(StressTestStressor.java:371)

              at org.radargun.stressors.StressTestStressor$Stressor.run(StressTestStressor.java:555)

      Caused by: SuspectedException

              at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:377)

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)

              ... 58 more

       

      Once the exception is caught, the performance of the system is drastically decreasing.

       

      The stess test details are,

      Duration                 = 500 minutes

      Attributes               = 4000000

      Read-write radio     = 6:4

      Accessing treads   = 40

       

      Attached infinispan and jgroup configuration.