ConcurrentModificationException during Context Invalidation
We are using Weld in a Webapplication running on JBoss 6.0, i.e. Weld 1.1.0 CR3 (on an AIX machine using J9 JVM, should that matter). After the newest update of our application we're suddenly seeing lots and lots of ConcurrentModificationExceptions in our logfiles, and I can't figure out why nor can I reproduce them. In previous version these never happened (using the same JBoss, Weld, JVM and OS). Here's the stacktrace:
12:00:00,384 ERROR [[/jazz]] Exception sending request destroyed lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:894) at java.util.HashMap$KeyIterator.next(HashMap.java:928) at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:170) at org.jboss.weld.context.AbstractManagedContext.deactivate(AbstractManagedContext.java:50) at org.jboss.weld.context.AbstractBoundContext.deactivate(AbstractBoundContext.java:82) at org.jboss.weld.servlet.WeldListener.requestDestroyed(WeldListener.java:115) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:204) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181) at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285) at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951) at java.lang.Thread.run(Thread.java:781)
You'll notice that this trace doesn't touch nor originates from parts of our application, which makes me scratch my head even more. In the new version we did significantly increase usage of CDI constructs (more SessionScoped beans, more injected beans, more bean resolution/lookups), which I'm guessing must have some role in it. Here's what I found out
- I can recreate the call stack: execution flows through that point when JBoss invalidates a session due to session timeouts.
- On the production system the error happens multiple times per second. There's around ~200 concurrent, active users online at the same time and around 400 concurrent sessions.
- I can't trigger this CME locally, even when running load tests using JMeter with hundreds of threads.
- Our own CDI session handling (see below) does pass through the same method "AbstractContext.destroy", but from a different origin.
CDI usage in our application is mostly standard: lots of session scoped beans (some in other scopes). One special case we do as a work around is controlling custom CDI session within threads that we start. Contexts are associated, activated and afterwards cleaned up in a finally block:
BoundSessionContext cdiContext= null;
Map<String, Object> cdiDataStore= new HashMap<String, Object>();
try {
cdiContext= getReference(BoundSessionContext.class, BoundLiteral.INSTANCE);
cdiContext.associate(cdiDataStore);
cdiContext.activate();
// does the main work
runnable.run();
} finally {
if (cdiContext != null) {
try {
cdiContext.invalidate();
cdiContext.deactivate();
} finally {
cdiContext.dissociate(cdiDataStore);
}
}
}
I know it's ugly, but it works. Well, seems to. It does clean up the context that it starts, so it shouldn't cause any problems, or does it?
Here's what I'm looking for:
- Does anyone ever encountered something like this CME?
- Any ideas as to why Weld is running into this modification exception?
- Did and if so how cause our code this? How can I cause the concurrent access deep within Weld? How can JBoss' session invalidation cause this?
