Cluster is very unstable
Hi,
I am running 2 node infinispan UDP cluster and used RadarGun to validate the performance of the data grid.
The cluster is running good with smaller load. If the load is increased, cluster becoming very unstable.
The nodes are becoming into suspect mode and disconnecting very often form the cluster which leads frequent cluster reforming.
05:07:36,468 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 16729 ms, adding it to suspect list
05:07:36,469 DEBUG [FD_ALL] suspecting [localhost-51593]
05:07:38,472 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 18733 ms, adding it to suspect list
05:07:38,473 DEBUG [FD_ALL] suspecting [localhost-51593]
05:07:40,474 DEBUG [FD_ALL] haven't received a heartbeat from localhost-51593 for 20735 ms, adding it to suspect list
05:07:40,474 DEBUG [FD_ALL] suspecting [localhost-51593]
05:07:41,472 DEBUG [RequestCorrelator] suspect=localhost-51593
05:07:41,478 ERROR [InvocationContextInterceptor] ISPN000136: Execution error
org.infinispan.remoting.transport.jgroups.SuspectException: Node localhost-51593 was suspected
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:184)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:224)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:249)
at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:236)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1196)
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783)
at org.infinispan.CacheImpl.put(CacheImpl.java:777)
at org.infinispan.CacheImpl.put(CacheImpl.java:771)
at org.infinispan.CacheSupport.put(CacheSupport.java:53)
at org.radargun.cachewrappers.InfinispanWrapper.put(InfinispanWrapper.java:244)
at org.radargun.cachewrappers.InfinispanExplicitLockingWrapper.put(InfinispanExplicitLockingWrapper.java:72)
at org.radargun.stressors.StressTestStressor$FixedSetSharedOperationLogic.init(StressTestStressor.java:371)
at org.radargun.stressors.StressTestStressor$Stressor.run(StressTestStressor.java:555)
Caused by: SuspectedException
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:377)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
... 58 more
Once the exception is caught, the performance of the system is drastically decreasing.
The stess test details are,
Duration = 500 minutes
Attributes = 4000000
Read-write radio = 6:4
Accessing treads = 40
Attached infinispan and jgroup configuration.
-
jgroups.xml 2.5 KB
-
infinispan-configuration.xml 659 bytes
