7 Replies Latest reply on Dec 3, 2007 1:48 AM by Scott Stark

    ConcurrentModificationException / Classloading / Deployment

    Andrew Rubinger Master

      While deploying some MDB Unit Tests today, came across the following error (sporadically):

      java.lang.RuntimeException: Failed to download and/or install client side AOP stack


      I've also seen other variations, reporting CNFE for at least 3 different classes (I've since thrown away these stack traces).

      The underlying cause is, however, consistent when the error is thrown:

      Caused by: java.util.ConcurrentModificationException
       at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1031)
       at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
       at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:235)
       at org.jboss.mx.loading.UnifiedClassLoader.loadClassImpl(UnifiedClassLoader.java:290)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:441)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$getClientAOPStack$aop(ClientConnectionFactoryDelegate.java:230)
       at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.getClientAOPStack(ClientConnectionFactoryDelegate.java)
       at org.jboss.jms.client.ClientAOPStackLoader.load(ClientAOPStackLoader.java:75)
       at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:192)


      I've been duplicating this by running the "ejbthree1123" test a few times, usually I can get it to pop up. But there's at least a couple threads accessing/writing to one Map instance in there somewhere. :)

      S,
      ALR

        • 1. Re: ConcurrentModificationException / Classloading / Deploym
          Andrew Rubinger Master

          If some more context is helpful:

          To duplicate:

          shell$> cd [jboss working copy]/ejb3
          shell$> ./build.sh -f build-test.xml -Dtest=ejbthree1123 ejbthree1123 test


          Another example error:
          java.lang.NoClassDefFoundError: org/jboss/resource/adapter/jms/inflow/dlq/AbstractDLQHandler


          Cause:
          java.util.ConcurrentModificationException
           at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1031)
           at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
           at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:235)
           at org.jboss.mx.loading.UnifiedClassLoader.loadClassImpl(UnifiedClassLoader.java:290)
           at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:441)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
           at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
           at java.lang.Class.getDeclaredConstructors0(Native Method)
           at java.lang.Class.privateGetDeclaredConstructors(Class.java:2357)
           at java.lang.Class.getConstructor0(Class.java:2671)
           at java.lang.Class.newInstance0(Class.java:321)
           at java.lang.Class.newInstance(Class.java:303)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:400)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:339)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:692)
           at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
           at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
           at java.lang.Thread.run(Thread.java:595)


          S,
          ALR

          • 2. Re: ConcurrentModificationException / Classloading / Deploym
            Scott Stark Master

            New class loaders for the package can be added while someone is looking up a class. This will need to be changed to a copy on write type of logic. I have made that change.

            • 3. Re: ConcurrentModificationException / Classloading / Deploym
              Andrew Rubinger Master

              Thanks, Scott. Brought in your change; problem still persists, though. Reliably reproduced through the instructions below.

              S,
              ALR

              • 5. Re: ConcurrentModificationException / Classloading / Deploym
                Scott Stark Master

                Its the TreeSets in the ConcurrentReaderHashMap that are being used and seeing the modifications. I have further updated the pkgSet copy on write updates for removal of class loaders and no longer seen any ConcurrentModificationException when running the test. The deployment is being torn down while the jms endpoint is still starting up so there are still race condition problems with the test.

                • 6. Re: ConcurrentModificationException / Classloading / Deploym
                  Andrew Rubinger Master

                  That one'll do 'er.

                  I've updated the example test to include some logic which includes a wait (as this is an async call), which doesn't kill the race condition, but is enough to ensure the hare beats the tortoise 99% of the time.

                  Don't run it twice though without server restart, may have uncovered another classloading issue on uninstall/install?

                  ERROR [AbstractKernelController] Error installing to Real: name=vfsfile:/home/alrubinger/business/jboss/wc/jbossas/trunk/ejb3/output/test-lib/ejbthree1123.jar state=PostClassLoader mode=Manual requiredState=Real
                  org.jboss.deployers.spi.DeploymentException: Error deploying ejbthree1123.jar: Error creating ejb container TestStatelessBean: java.lang.ClassNotFoundException: Invalid use of destroyed classloader, UCL destroyed at:
                   at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:158)


                  S,
                  ALR

                  • 7. Re: ConcurrentModificationException / Classloading / Deploym
                    Scott Stark Master

                    Yes, the removal of the class loader after the copy on write change was not happening. Its been corrected.