HTTP session replication ConcurrentModificationException

[koreth]

I have a stress test tool that does lots of JSP fetches without retaining any cookies between requests. The container therefore is creating a new session for each request (which is the expected behavior) and my JSP code is setting a few session attributes when it sees it's dealing with a new session.

On a standalone server, this works fine. Once I'm running in a cluster with synchronous HTTP session replication, I start getting lots of these:

23:59:59,284 ERROR [OrderedSynchronizationHandler] failed calling afterCompletion() on org.jboss.cache.interceptors.TransactionInterceptor$SynchronizationHandler@12f3f7
java.util.ConcurrentModificationException
 at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:448)
 at java.util.AbstractList$Itr.next(AbstractList.java:419)
 at java.util.AbstractCollection.toString(AbstractCollection.java:454)
 at java.lang.String.valueOf(String.java:2131)
 at java.lang.StringBuffer.append(StringBuffer.java:370)
 at org.jboss.cache.lock.IdentityLock.toString(IdentityLock.java:303)
 at java.lang.String.valueOf(String.java:2131)
 at java.lang.StringBuffer.append(StringBuffer.java:370)
 at org.jboss.cache.TreeCache.commit(TreeCache.java:2818)
 at org.jboss.cache.interceptors.TransactionInterceptor$SynchronizationHandler.afterCompletion(TransactionInterceptor.java:108)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.afterCompletion(OrderedSynchronizationHandler.java:79)
 at org.jboss.tm.TransactionImpl.doAfterCompletion(TransactionImpl.java:1418)
 at org.jboss.tm.TransactionImpl.completeTransaction(TransactionImpl.java:1090)
 at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:349)
 at org.jboss.tm.TxManager.commit(TxManager.java:200)
 at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:126)
 at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:364)
 at org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:213)
 at org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:37)
 at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
 at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
 at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
 at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
 at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
 at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
 at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
 at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300)
 at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374)
 at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
 at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:675)
 at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
 at java.lang.Thread.run(Thread.java:534)

It looks like maybe the fact that I'm setting session attributes while another thread is creating a new session is causing the session replication code to get confused.

Anyone else seen this behavior?

[koreth]

Of course, the server has no control over how many concurrent requests potential clients will issue. You can however configure tomcat how many requests it should handle concurrently.

 <Connector port="8080" address="${jboss.bind.address}"
 maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
 enableLookups="false" redirectPort="8443" acceptCount="100"
 connectionTimeout="20000" disableUploadTimeout="true"/>

[koreth]

I should add that switching to asynchronous instant replication, or to interval-based replication, does not seem to help.

[belaban]

Seems to be a problem in IdentityLock: read_owners are updated before toString() is done iterating through the list. I have fixed this in CVS head.
You can backport the toString() to you 1.1.1 version and see whether this works (it should).

Bela