2 Replies Latest reply on May 6, 2014 3:17 PM by mister_s

    ConcurrentModificationException during Context Invalidation

    mister_s Newbie

      We are using Weld in a Webapplication running on JBoss 6.0, i.e. Weld 1.1.0 CR3 (on an AIX machine using J9 JVM, should that matter). After the newest update of our application we're suddenly seeing lots and lots of ConcurrentModificationExceptions in our logfiles, and I can't figure out why nor can I reproduce them. In previous version these never happened (using the same JBoss, Weld, JVM and OS). Here's the stacktrace:

       

      12:00:00,384 ERROR [[/jazz]] Exception sending request destroyed lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener
      java.util.ConcurrentModificationException
      at java.util.HashMap$HashIterator.nextEntry(HashMap.java:894)
      at java.util.HashMap$KeyIterator.next(HashMap.java:928)
      at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:170)
      at org.jboss.weld.context.AbstractManagedContext.deactivate(AbstractManagedContext.java:50)
      at org.jboss.weld.context.AbstractBoundContext.deactivate(AbstractBoundContext.java:82)
      at org.jboss.weld.servlet.WeldListener.requestDestroyed(WeldListener.java:115)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:204)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
      at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
      at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
      at java.lang.Thread.run(Thread.java:781)
      
      

       

      You'll notice that this trace doesn't touch nor originates from parts of our application, which makes me scratch my head even more. In the new version we did significantly increase usage of CDI constructs (more SessionScoped beans, more injected beans, more bean resolution/lookups), which I'm guessing must have some role in it. Here's what I found out

       

      • I can recreate the call stack: execution flows through that point when JBoss invalidates a session due to session timeouts.
      • On the production system the error happens multiple times per second. There's around ~200 concurrent, active users online at the same time and around 400 concurrent sessions.
      • I can't trigger this CME locally, even when running load tests using JMeter with hundreds of threads.
      • Our own CDI session handling (see below) does pass through the same method "AbstractContext.destroy", but from a different origin.

       

      CDI usage in our application is mostly standard: lots of session scoped beans (some in other scopes). One special case we do as a work around is controlling custom CDI session within threads that we start. Contexts are associated, activated and afterwards cleaned up in a finally block:

       

      BoundSessionContext cdiContext= null;
      Map<String, Object> cdiDataStore= new HashMap<String, Object>();
      
      
      try {
          cdiContext= getReference(BoundSessionContext.class, BoundLiteral.INSTANCE);
          cdiContext.associate(cdiDataStore);
          cdiContext.activate();
      
      
          // does the main work
          runnable.run();
      } finally {
          if (cdiContext != null) {
              try {
                  cdiContext.invalidate();
                  cdiContext.deactivate();
              } finally {
                  cdiContext.dissociate(cdiDataStore);
              }
          }
      }
      
      

       

      I know it's ugly, but it works. Well, seems to. It does clean up the context that it starts, so it shouldn't cause any problems, or does it?

       

      Here's what I'm looking for:

      • Does anyone ever encountered something like this CME?
      • Any ideas as to why Weld is running into this modification exception?
      • Did and if so how cause our code this? How can I cause the concurrent access deep within Weld? How can JBoss' session invalidation cause this?
        • 1. Re: ConcurrentModificationException during Context Invalidation
          Martin Kouba Master

          Hi,

          first I would try to upgrade to AS7 or at least to some more recent version of Weld 1.1.x:

          export JBOSS_HOME=/path/to/your/AS6
          git clone git@github.com:weld/core.git
          cd core
          git checkout 1.1
          mvn clean install -DskipTests
          mvn clean package -Dweld.update.version=1.1.21.Final -f jboss-as/jboss-as-6/pom.xml
          

           

          Also I don't think your code around BoundSessionContext is related as the ConcurrentModificationException from your stack happens during the request context deactivation. The strange thing is you can't simulate this locally. Have you tried to simulate the session invalidation?

          • 2. Re: ConcurrentModificationException during Context Invalidation
            mister_s Newbie

            Hi Martin,

             

            Unfortunately updating to AS7 is out of the question, as much as I'd like too. I'm now trying your suggestion with Weld 1.1.

             

            In the meantime, we made some progress. We can reproduce the exception within our application, but I don't yet have a minimal sample reproducing it. Here's what we've found so far:

            * The exception only occurs on IBM's J9 VM

            * The exception only occurs if we enable our (custom) profiling component that tracks some metrics like request times, processing times, etc.

             

            As the name suggests our profiler does some processing around the actual request - but it still uses only "regular" CDI features. We currently suspect that the bug is triggered by some combination of Session- and RequestScoped (used by the profiler) beans.