7 Replies Latest reply on Dec 3, 2007 1:48 AM by starksm64

    ConcurrentModificationException / Classloading / Deployment

    alrubinger

      While deploying some MDB Unit Tests today, came across the following error (sporadically):

      java.lang.RuntimeException: Failed to download and/or install client side AOP stack


      I've also seen other variations, reporting CNFE for at least 3 different classes (I've since thrown away these stack traces).

      The underlying cause is, however, consistent when the error is thrown:

      Caused by: java.util.ConcurrentModificationException
       at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1031)
       at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
       at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:235)
       at org.jboss.mx.loading.UnifiedClassLoader.loadClassImpl(UnifiedClassLoader.java:290)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:441)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$getClientAOPStack$aop(ClientConnectionFactoryDelegate.java:230)
       at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.getClientAOPStack(ClientConnectionFactoryDelegate.java)
       at org.jboss.jms.client.ClientAOPStackLoader.load(ClientAOPStackLoader.java:75)
       at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:192)


      I've been duplicating this by running the "ejbthree1123" test a few times, usually I can get it to pop up. But there's at least a couple threads accessing/writing to one Map instance in there somewhere. :)

      S,
      ALR

        • 1. Re: ConcurrentModificationException / Classloading / Deploym
          alrubinger

          If some more context is helpful:

          To duplicate:

          shell$> cd [jboss working copy]/ejb3
          shell$> ./build.sh -f build-test.xml -Dtest=ejbthree1123 ejbthree1123 test


          Another example error:
          java.lang.NoClassDefFoundError: org/jboss/resource/adapter/jms/inflow/dlq/AbstractDLQHandler


          Cause:
          java.util.ConcurrentModificationException
           at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1031)
           at java.util.TreeMap$KeyIterator.next(TreeMap.java:1058)
           at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:235)
           at org.jboss.mx.loading.UnifiedClassLoader.loadClassImpl(UnifiedClassLoader.java:290)
           at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:441)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
           at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
           at java.lang.Class.getDeclaredConstructors0(Native Method)
           at java.lang.Class.privateGetDeclaredConstructors(Class.java:2357)
           at java.lang.Class.getConstructor0(Class.java:2671)
           at java.lang.Class.newInstance0(Class.java:321)
           at java.lang.Class.newInstance(Class.java:303)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation.setupDLQ(JmsActivation.java:400)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation.setup(JmsActivation.java:339)
           at org.jboss.resource.adapter.jms.inflow.JmsActivation$SetupActivation.run(JmsActivation.java:692)
           at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
           at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
           at java.lang.Thread.run(Thread.java:595)


          S,
          ALR

          • 2. Re: ConcurrentModificationException / Classloading / Deploym
            starksm64

            New class loaders for the package can be added while someone is looking up a class. This will need to be changed to a copy on write type of logic. I have made that change.

            • 3. Re: ConcurrentModificationException / Classloading / Deploym
              alrubinger

              Thanks, Scott. Brought in your change; problem still persists, though. Reliably reproduced through the instructions below.

              S,
              ALR

              • 4. Re: ConcurrentModificationException / Classloading / Deploym
                anil.saldhana

                Doesn't CopyOnWriteArrayList avoid ConcurrentModificationException by design?
                http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/CopyOnWriteArrayList.html

                It should.

                • 5. Re: ConcurrentModificationException / Classloading / Deploym
                  starksm64

                  Its the TreeSets in the ConcurrentReaderHashMap that are being used and seeing the modifications. I have further updated the pkgSet copy on write updates for removal of class loaders and no longer seen any ConcurrentModificationException when running the test. The deployment is being torn down while the jms endpoint is still starting up so there are still race condition problems with the test.

                  • 6. Re: ConcurrentModificationException / Classloading / Deploym
                    alrubinger

                    That one'll do 'er.

                    I've updated the example test to include some logic which includes a wait (as this is an async call), which doesn't kill the race condition, but is enough to ensure the hare beats the tortoise 99% of the time.

                    Don't run it twice though without server restart, may have uncovered another classloading issue on uninstall/install?

                    ERROR [AbstractKernelController] Error installing to Real: name=vfsfile:/home/alrubinger/business/jboss/wc/jbossas/trunk/ejb3/output/test-lib/ejbthree1123.jar state=PostClassLoader mode=Manual requiredState=Real
                    org.jboss.deployers.spi.DeploymentException: Error deploying ejbthree1123.jar: Error creating ejb container TestStatelessBean: java.lang.ClassNotFoundException: Invalid use of destroyed classloader, UCL destroyed at:
                     at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:158)


                    S,
                    ALR

                    • 7. Re: ConcurrentModificationException / Classloading / Deploym
                      starksm64

                      Yes, the removal of the class loader after the copy on write change was not happening. Its been corrected.