0 Replies Latest reply on Dec 15, 2015 9:29 AM by Neji Gammoudi

    [Cluster-UDP]: Cluster is not stable with a massive load

    Neji Gammoudi Newbie

      Hello,

      I am running 2 exoplatform tomcat nodes with cluster using the default UDP mode and use a jmeter script to validate the login performance.

       

      The cluster is running good with smaller load but when the load is increased the cluster become unstable and i got a TimeoutException of jgroups , below the stack-trace:

       

      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]

      Caused by: org.jgroups.TimeoutException: timeout sending message to exo-node1-1821

      at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390) ~[jgroups-3.2.7.Final.jar:3.2.7.Final]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) ~[infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ~[infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      ... 91 common frames omitted

      2015-12-15 15:20:23,720 | WARN  | ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[exo-node2-3705, exo-node1-1821], isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=[ChangesKey-a602dadbc0a802ec504f633177cf6b64-null], topologyId=2, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@10f3f0} org.infinispan.transaction.synchronization.SynchronizationAdapter@10f40f with exception [com.arjuna.ats.jta<Forum-Thread>]

      org.infinispan.CacheException: Could not commit.

      at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83) ~[infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) ~[jbossjta-4.16.6.Final.jar:na]
      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) ~[jbossjta-4.16.6.Final.jar:na]
      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) ~[jbossjta-4.16.6.Final.jar:na]
      at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) ~[jbossjta-4.16.6.Final.jar:na]
      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) [jbossjta-4.16.6.Final.jar:na]
      at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) [jbossjta-4.16.6.Final.jar:na]
      at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1196) [infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783) [infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at org.infinispan.CacheImpl.put(CacheImpl.java:777) [infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at org.infinispan.DecoratedCache.put(DecoratedCache.java:320) [infinispan-core-5.2.9.Final.jar:5.2.9.Final]
      at org.exoplatform.services.jcr.impl.core.query.ispn.ISPNIndexChangesFilter.doUpdateIndex(ISPNIndexChangesFilter.java:172) [exo.jcr.component.core-1.16.4-GA.jar:1.16.4-GA]
      at org.exoplatform.services.jcr.impl.core.query.IndexerChangesFilter.doUpdateIndex(IndexerChangesFilter.java:231) [exo.jcr.component.core-1.16.4-GA.jar:1.16.4-GA]
      at org.exoplatform.services.jcr.impl.core.query.IndexerChangesFilter.onSaveItems(IndexerChangesFilter.java:160) [exo.jcr.component.core-1.16.4-GA.jar:1.16.4-GA]
      at org.exoplatform.services.jcr.impl.core.query.SearchManager.onSaveItems(SearchManager.java:640) [exo.jcr.component.core-1.16.4-GA.jar:1.16.4-GA]
      at org.exoplatform.services.jcr.impl.dataflow.persistent.WorkspacePersistentDataManager.notifySaveItems(WorkspacePersistentDataManager.java:1300) [exo.jcr.component.core-1.16.4-GA.jar:1.16.

       

      The infinispan version is : 5.2.9.Final

      The jgroups version is : 3.2.7.Final

      In attachment the jgroups configuration file used in our application.

      Is there any recommended config to use in order to fix this problem?

      Regards