Getting lots of exception related to infinispan...| JBoss.org Content Archive (Read Only)

15. Re: Re: Re: Re: Getting lots of exception related to infinispan while load testing

johnhpatton May 28, 2014 5:42 PM (in response to johnhpatton)

I've added keep_alive_time, which seems to have reduced the number of CMEs, but we're still seeing one every so often. No 500s anymore, tho, so perhaps it's not really an "ERROR" but noise?

                   <transport type="TCP" socket-binding="jgroups-tcp">

                        <property name="oob_thread_pool.max_threads">300</property>

                       <property name="oob_thread_pool.keep_alive_time">20000</property>

                    </transport>

Update: Took a while, but we finally were able to get some more exceptions:

[Server:WILDFLYNODE1] 16:36:44,143 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-193) ISPN000136: Execution error: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)

[Server:WILDFLYNODE1]   at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:581)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:141)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:524)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:219)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:203)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:83)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:114)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:101)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:194)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:94)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:66)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:154)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:58)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:263)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:312)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:69)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:101)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:83)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:64)

[Server:WILDFLYNODE1]   at org.infinispan.CacheImpl.endBatch(CacheImpl.java:777)

[Server:WILDFLYNODE1]   at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:53)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.end(InfinispanBatcher.java:56)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.close(InfinispanBatcher.java:46)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:72)

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:704) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:522) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.Connectors.executeRootHandler(Connectors.java:168) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1] Caused by: java.lang.RuntimeException: Failure to marshal argument(s)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:333)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:132)

[Server:WILDFLYNODE1]   ... 76 more

[Server:WILDFLYNODE1] Caused by: java.util.ConcurrentModificationException

[Server:WILDFLYNODE1]   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.HashMap$EntryIterator.next(HashMap.java:966) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.HashMap$EntryIterator.next(HashMap.java:964) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:681)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:77)

[Server:WILDFLYNODE1]   at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:150)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:876)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:42)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:30)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:148)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:116)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:100)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:59)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:148)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:74)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:331)

[Server:WILDFLYNODE1]   ... 78 more

[Server:WILDFLYNODE1] Caused by: an exception which occurred:

[Server:WILDFLYNODE1]   in object java.util.HashMap@d82b7fb9

[Server:WILDFLYNODE1]   in object org.jboss.as.clustering.marshalling.SimpleMarshalledValue@d82b7fb9

[Server:WILDFLYNODE1]   in object org.infinispan.commands.write.ReplaceCommand@80a4d6fb

[Server:WILDFLYNODE1]   in object org.infinispan.commands.tx.PrepareCommand@417d78de

Followed by:

[Server:WILDFLYNODE1] 16:36:44,158 ERROR [org.infinispan.transaction.TransactionCoordinator] (default task-193) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)

[Server:WILDFLYNODE1]   at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:581)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:141)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:524)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:219)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:203)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:78)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:83)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:114)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:101)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:194)

[Server:WILDFLYNODE1]   at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:94)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:66)

[Server:WILDFLYNODE1]   at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:96)

[Server:WILDFLYNODE1]   at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:125)

[Server:WILDFLYNODE1]   at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:154)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:58)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:263)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:312)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:69)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:101)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:83)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:64)

[Server:WILDFLYNODE1]   at org.infinispan.CacheImpl.endBatch(CacheImpl.java:777)

[Server:WILDFLYNODE1]   at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:53)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.end(InfinispanBatcher.java:56)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.close(InfinispanBatcher.java:46)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:72)

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:704) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:522) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.Connectors.executeRootHandler(Connectors.java:168) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1] Caused by: java.lang.RuntimeException: Failure to marshal argument(s)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:333)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:132)

[Server:WILDFLYNODE1]   ... 76 more

[Server:WILDFLYNODE1] Caused by: java.util.ConcurrentModificationException

[Server:WILDFLYNODE1]   at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.HashMap$EntryIterator.next(HashMap.java:966) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.HashMap$EntryIterator.next(HashMap.java:964) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:681)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:77)

[Server:WILDFLYNODE1]   at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:150)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:876)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:42)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:30)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:148)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:116)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:100)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:59)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:148)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

[Server:WILDFLYNODE1]   at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:74)

[Server:WILDFLYNODE1]   at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

[Server:WILDFLYNODE1]   at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

[Server:WILDFLYNODE1]   at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:331)

[Server:WILDFLYNODE1]   ... 78 more

[Server:WILDFLYNODE1] Caused by: an exception which occurred:

[Server:WILDFLYNODE1]   in object java.util.HashMap@d82b7fb9

[Server:WILDFLYNODE1]   in object org.jboss.as.clustering.marshalling.SimpleMarshalledValue@d82b7fb9

[Server:WILDFLYNODE1]   in object org.infinispan.commands.write.ReplaceCommand@80a4d6fb

[Server:WILDFLYNODE1]   in object org.infinispan.commands.tx.PrepareCommand@417d78de

Followed by:

[Server:WILDFLYNODE1] 16:36:44,171 ERROR [org.infinispan.transaction.tm.DummyTransaction] (default task-193) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=17, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@3c7c} org.infinispan.transaction.synchronization.SynchronizationAdapter@3c9b: org.infinispan.commons.CacheException: Could not commit.

[Server:WILDFLYNODE1]   at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:60)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:263)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:312)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:69)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:80)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:101)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:83)

[Server:WILDFLYNODE1]   at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:64)

[Server:WILDFLYNODE1]   at org.infinispan.CacheImpl.endBatch(CacheImpl.java:777)

[Server:WILDFLYNODE1]   at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:53)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.end(InfinispanBatcher.java:56)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.infinispan.InfinispanBatcher$1.close(InfinispanBatcher.java:46)

[Server:WILDFLYNODE1]   at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:72)

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:704) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:522) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.0.Final.jar:1.0.0.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.Connectors.executeRootHandler(Connectors.java:168) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.4.Final.jar:1.0.4.Final]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1]   at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:WILDFLYNODE1] Caused by: javax.transaction.xa.XAException

[Server:WILDFLYNODE1]   at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:204)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:156)

[Server:WILDFLYNODE1]   at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:58)

[Server:WILDFLYNODE1]   ... 23 more

Pattern repeats until I restart.

16. Re: Getting lots of exception related to infinispan while load testing

pferraro May 29, 2014 10:44 AM (in response to johnhpatton)

Are you certain that your load test does not trigger multiple concurrent/overlapping requests for the same session?

Do you have the option to upgrade to WildFly 8.1? 8.0 does not completely prevent concurrent access to a session - while 8.1 does.

17. Re: Getting lots of exception related to infinispan while load testing

johnhpatton May 29, 2014 2:33 PM (in response to pferraro)

Yep, confirmed. We are not triggering multiple concurrent/overlapping requests for the same session. They're using loadrunner which handles each user connection seperately, all requests for each user are sequential.

We do have the option, but there's a concern about upgrading prior to final release here. Release candidates are not on the table at this time.

18. Re: Getting lots of exception related to infinispan while load testing

pferraro May 29, 2014 3:44 PM (in response to johnhpatton)

The 8.1.0.Final release is expected to be released quite soon.

I asked previously about the network. I see that you are using the TCP-based stack - is that because multicast is disabled on your network? If so, did you make the configuration changes to NAKACK2 that I mentioned here?

Re: Re: Re: Re: Re: Re: Getting lots of exception related to infinispan while load testing

19. Re: Getting lots of exception related to infinispan while load testing

johnhpatton May 29, 2014 4:16 PM (in response to pferraro)

Yep, totally forgot to respond to that. Yes, we have multicast disabled between server racks. Each rack is in its own subnet, each app spans multiple racks.

As for the config, here's what we implemented in the web cache container:

                <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                    <transport stack="tcp" cluster="${jboss.cluster.group.name}" lock-timeout="240000"/>

                    <distributed-cache name="dist" batching="true" mode="SYNC" owners="3" l1-lifespan="0">

                        <state-transfer timeout="240000"/>

                        <locking acquire-timeout="120000"/>

                        <file-store/>

                    </distributed-cache>

                </cache-container>

And here's what we implemented in the tcp stack:

                <stack name="tcp">

                    <transport type="TCP" socket-binding="jgroups-tcp">

                        <property name="oob_thread_pool.max_threads">

                            300

                        </property>

                        <property name="oob_thread_pool.keep_alive_time">

                            20000

                        </property>

                    </transport>

                    <protocol type="TCPPING">

                        <property name="initial_hosts">

                            ${jgroups.ha.tcpping.initial_hosts}

                        </property>

                        <property name="num_initial_members">

                            ${jgroups.ha.tcpping.num_initial_members}

                        </property>

                        <property name="timeout">

                            ${jgroups.ha.tcpping.timeout}

                        </property>

                    </protocol>

                    <protocol type="MERGE2"/>

                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                    <protocol type="FD"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2">

                        <property name="use_mcast_xmit">

                            false

                        </property>

                        <property name="use_mcast_xmit_req">

                            false

                        </property>

                    </protocol>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG2"/>

                    <protocol type="RSVP"/>

                </stack>

20. Re: Getting lots of exception related to infinispan while load testing

johnhpatton May 30, 2014 2:27 PM (in response to johnhpatton)

We've managed to get rid of all "ERROR" CMEs!! However, we still have an occasional WARN show up. Here's the current config:

In the urn:jboss:domain:batch:1.0 subsystem, we reduced the thread pool's max-threads from 160 down to 10 (i don't recall why we set it to 160):

                <thread-pool>

                    <max-threads count="10"/>

                    <keepalive-time time="100" unit="milliseconds"/>

                </thread-pool>

In the urn:jboss:domain:ejb3:2.0 subsystem, we reduced the thread pool's max-threads from 160 down to 10 (again, I don't recall why we set it to 160):

                <thread-pools>

                    <thread-pool name="default">

                        <max-threads count="10"/>

                        <keepalive-time time="100" unit="milliseconds"/>

                    </thread-pool>

                </thread-pools>

Infinispan subsystem, the meat of our changes:

            <subsystem xmlns="urn:jboss:domain:infinispan:2.0">

                <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">

                    <transport lock-timeout="60000"/>

                    <replicated-cache name="default" batching="true" mode="SYNC">

                        <locking isolation="REPEATABLE_READ"/>

                    </replicated-cache>

                </cache-container>

                <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                    <transport stack="tcp" cluster="${jboss.cluster.group.name}" lock-timeout="300000"/>

                    <distributed-cache name="dist" start="EAGER" batching="true" mode="SYNC" remote-timeout="300000" owners="2" l1-lifespan="0">

                        <locking acquire-timeout="60000" concurrency-level="3000" striping="false"/>

                        <eviction strategy="LIRS" max-entries="1000"/>

                        <state-transfer timeout="300000"/>

                        <file-store shared="true" preload="true"/>

                    </distributed-cache>

                </cache-container>

            </subsystem>

Jgroups subsystem, unchanged from prior testing, added here for completeness:

            <subsystem xmlns="urn:jboss:domain:jgroups:2.0" default-stack="tcp">

               <stack name="tcp">

                    <transport type="TCP" socket-binding="jgroups-tcp">

                        <property name="oob_thread_pool.max_threads">

                            300

                        </property>

                        <property name="oob_thread_pool.keep_alive_time">

                            20000

                        </property>

                    </transport>

                    <protocol type="TCPPING">

                        <property name="initial_hosts">

                            ${jgroups.ha.tcpping.initial_hosts}

                        </property>

                        <property name="num_initial_members">

                            ${jgroups.ha.tcpping.num_initial_members}

                        </property>

                        <property name="timeout">

                            ${jgroups.ha.tcpping.timeout}

                        </property>

                    </protocol>

                    <protocol type="MERGE2"/>

                    <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                    <protocol type="FD"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2">

                        <property name="use_mcast_xmit">

                            false

                        </property>

                        <property name="use_mcast_xmit_req">

                            false

                        </property>

                    </protocol>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG2"/>

                    <protocol type="RSVP"/>

                </stack>

            </subsystem>

This is the WARN we're receiving:

[Server:WILDFLYNODE1] 09:53:21,882 WARN [org.infinispan.eviction.PassivationManagerImpl] (default task-273) ISPN000028: Unable to passivate entry under J8YAiKcUZjsagmnoBl0e4iaE: org.infinispan.persistence.spi.PersistenceException: org.infinispan.persistence.spi.PersistenceException: java.util.ConcurrentModificationException

This appears to be alright, tho, as it's not resulting in any 500 responses. Would love to figure out what attribute to twiddle to reduce/remove these, tho.

21. Re: Getting lots of exception related to infinispan while load testing

pferraro Jun 2, 2014 8:27 AM (in response to johnhpatton)

What value have you defined for <max-active-sessions/> in your jboss-web.xml?

Passivation of web session happens when the number of sessions managed by a node exceeds this limit. If we're in the process of passivating a session, and the session is concurrently accessed, this exception can be thrown by the passivation thread (although this is fixed in 8.1.0.Final - which was released over the weekend, btw).

If you don't need passivation, you can omit this property from jboss-web.xml - and the exception will go away.

22. Re: Getting lots of exception related to infinispan while load testing

rishikesh_darandale Jun 2, 2014 8:53 AM (in response to pferraro)

Thanks Paul for all your help on this.

We are not using <max-active-sessions/> parameter in jboss-web.xml, hence it should be picking it's default value.

23. Re: Getting lots of exception related to infinispan while load testing

pferraro Jun 2, 2014 4:16 PM (in response to rishikesh_darandale)

OK - I see what's going on. You've configured your infinispan cache with <eviction/> settings. This is not the recommended way to configure web session passivation. Instead you should configure your jboss-web.xml with <max-active-sessions>...</max-active-sessions> based on the maximum number of sessions that should be kept in memory at any given time. Also, why did you configure your <file-store/> with shared="true"?

24. Re: Re: Getting lots of exception related to infinispan while load testing

rishikesh_darandale Jun 3, 2014 8:55 AM (in response to pferraro)

Thanks Paul.

Do you mean that instead of <eviction/> settings, we should use below kind of config in jboss-web.xml for session passivation? But currently, its not configured that means it is off. correct?

   <max-active-sessions>20</max-active-sessions>
   <passivation-config>
     <use-session-passivation>true</use-session-passivation>
     <passivation-min-idle-time>60</passivation-min-idle-time>
     <passivation-max-idle-time>600</passivation-max-idle-time>
   </passivation-config>

We have used shared=true because, we are testing our application under load with fail over and continuously getting CMEs. Hence we tried this option setting to true to resolve CMEs and that worked.

25. Re: Getting lots of exception related to infinispan while load testing

pferraro Jun 3, 2014 5:01 PM (in response to rishikesh_darandale)

Not quite. The <passivation-config/> element and its child elements are deprecated in WildFly. If specified, they will be ignored. WildFly configures passivation based soley on <max-active-sessions/>. If no value is specified for <max-active-sessions/>, then the assumption is that you can have an infinite number of active sessions, thus the web session clustering logic will never passivate sessions.

Alternatively, you omit <max-active-sessions/> and instead let infinispan manage the passivation of cache entries via <eviction max-entries="..."/>. The caveats here are:

There is not a 1:1 relationship between a web session and a cache entry. A web application can be configured to use either SESSION or ATTRIBUTE replication granularity. If using SESSION granularity (the default), each session corresponds to 2 cache entries, one containing the session's meta data, and the other containing the map of session attributes. If using ATTRIBUTE granularity, each session corresponds to N+1 cache entries, where N is the number of session attributes. Thus the value for <eviction max-entries="..."/> should be defined accordingly.
Because infinispan doesn't passivate all cache entries per web session at the same time, application that rely on passivation/activation listeners should not use infinispan-managed eviction.

26. Re: Getting lots of exception related to infinispan while load testing

rituraj Jun 4, 2014 7:04 AM (in response to pferraro)

Hi Paul,

Thanks for the detailed explanation ...i have another question here ...

There is not a 1:1 relationship between a web session and a cache entry. A web application can be configured to use either SESSION or ATTRIBUTE replication granularity. If using SESSION granularity (the default), each session corresponds to 2 cache entries, one containing the session's meta data, and the other containing the map of session attributes. If using ATTRIBUTE granularity, each session corresponds to N+1 cache entries, where N is the number of session attributes. Thus the value for <eviction max-entries="..."/> should be defined accordingly.

we are trying to get the correct value assigned to the <eviction max-entries="..."/> ....now my question is whether the value defined in the eviction max-entries is wrt to
the cluster or node/instance specific ...

i think it should be node specific but wanted to confirm...

another question is related to stores ...there are 4 stores altogether as "store" "jdbc-store" "file-store" and "remote-store"...in which file-store is being used as default ...what is the significance of the different stores and why are we using file-store as default ...also we make shared=true because of the CMEs we were getting ...it looks
like that change has made it possible ...also there is passivation=true in the file-store ...is the passivation coming into picture from here ...
but wanted to know the best way of using it ...

27. Re: Getting lots of exception related to infinispan while load testing

pferraro Jun 4, 2014 8:03 PM (in response to rituraj)

we are trying to get the correct value assigned to the <eviction max-entries="..."/> ....now my question is whether the value defined in the eviction max-entries is wrt to

the cluster or node/instance specific ...

This value is node specific, although it includes both entries that are owned by the node and the backup copies of entries owned by remote nodes, which, on average, should be N*M, where N is the number of owners and M is the number of sessions owned by a given node.

another question is related to stores ...there are 4 stores altogether as "store" "jdbc-store" "file-store" and "remote-store"...in which file-store is being used as default ...what is the significance of the different stores and why are we using file-store as default

There are 6 different supported cache stores, actually. 3 are jdbc-based cache stores for database storage. One for string-based cache keys, one for non-string based cache keys, and another for mixed string and non-string cache keys. The remote-store uses the hotrod protocol to store cache entries in a remote Infinispan cache. "store" is a generic configuration option which can refer to any implementation of org.infinispan.configuration.cache.StoreConfigurationBuilder. file-store is the used in the WildFly default configuration because requires no external resources apart from the filesystem.

also we make shared=true because of the CMEs we were getting ...it looks like that change has made it possible

shared="true" adds cluster-wide locking whenever a session is persisted to your cache store. You can probably imagine how this would be necessary if the cache store was a shared resource among all of your nodes. To be honest, I still don't understand what is causing these CMEs. You should not encounter this when passivation is configured correctly, i.e. using <max-active-sessions/> via jboss-web.xml instead of <eviction max-entries="..."/> in standalone-ha.xml. Can you consider upgrading to 8.1.0.Final?

also there is passivation=true in the file-store ...is the passivation coming into picture from here ...

but wanted to know the best way of using it ...

When a cache store is configured with passivation="false", then all cache entries will be written to the store. The number of cache entries in memory will be the same as the number of cache entries in the store. When passivation="true" (the default in WildFly), the cache store is only ever used to persist excess sessions that are evicted from memory. Thus a given cache entry is only ever in memory or on disk, but never both. passivation="false" is not particularly useful for clustered caches, since backup copies of cache entries are store in memory on remote nodes. The primary use case is local caches, where the persistent store is the only backup location.

28. Re: Getting lots of exception related to infinispan while load testing

rituraj Jun 4, 2014 12:33 PM (in response to pferraro)

Paul,

Thanks for replying and explaining all this ...we will work on your recommendations for all those values and get back to you on this ....

Thanks again...

29. Re: Getting lots of exception related to infinispan while load testing

pferraro Jun 6, 2014 2:00 PM (in response to rituraj)

FYI - I've opened [WFLY-3465] Using Infinispan eviction manager can cause ConcurrentModificationException during web session access - JBos…

... to improve the compatibility of web session clustering with infinispan's eviction manager.