5 Replies Latest reply on Sep 22, 2009 8:18 AM by adrian.brock

    Potential deadlock in ClassLoaderManager

    thomas.diesler

      This code intermittently deadlocks

      ClassLoaderManager:126

       synchronized (toTaskList)
       {
       toTaskList.add(0, threadTask);
       loadTask.nextEvent();
       toTaskList.notify();
       }
      


      I sometimes see this when running the JBoss OSGi HttpServiceTestCase. The Apache HttpService impl seems to load classes from its own thread.

      "Jetty HTTP Service Launcher" prio=10 tid=0x821f8c00 nid=0x4447 runnable [0x81bfa000]
       java.lang.Thread.State: RUNNABLE
       at java.util.Collections$SynchronizedList.add(Collections.java:1822)
       - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)
       at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:128)
       - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)
       - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)
       at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1082)
       - locked <0x950741b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:808)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x950741b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x950741b0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.Class.forName0(Native Method)
       at java.lang.Class.forName(Class.java:247)
       at org.jboss.classloader.plugins.loader.ClassLoaderToLoaderAdapter.loadClass(ClassLoaderToLoaderAdapter.java:172)
       at org.jboss.classloader.spi.ClassLoaderDomain.loadClassFromParent(ClassLoaderDomain.java:352)
       at org.jboss.classloader.spi.ClassLoaderDomain.loadClassBefore(ClassLoaderDomain.java:307)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:251)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:575)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:533)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:510)
       at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
       at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
       at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
       at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
       at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:575)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:533)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:510)
       at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
       at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
       at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
       at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x95409ce8> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.apache.felix.http.jetty.Activator.initializeHTTP(Activator.java:403)
       at org.apache.felix.http.jetty.Activator.initializeJetty(Activator.java:373)
       at org.apache.felix.http.jetty.Activator.startJetty(Activator.java:205)
       at org.apache.felix.http.jetty.Activator.run(Activator.java:248)
       at java.lang.Thread.run(Thread.java:619)
      
      "HDScanner" prio=10 tid=0x81940400 nid=0x4446 waiting on condition [0x81f5c000]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x953b61b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "JBoss System Threads(1)-1" daemon prio=10 tid=0x82aac000 nid=0x4445 runnable [0x81fad000]
       java.lang.Thread.State: RUNNABLE
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
       - locked <0x95341158> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:453)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at org.jnp.server.Main$AcceptHandler.run(Main.java:597)
       at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "GC Daemon" daemon prio=10 tid=0x82143c00 nid=0x4444 in Object.wait() [0x81ffe000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x9505fda8> (a sun.misc.GC$LatencyLock)
       at sun.misc.GC$Daemon.run(GC.java:100)
       - locked <0x9505fda8> (a sun.misc.GC$LatencyLock)
      
      "RMI Reaper" prio=10 tid=0x815f7800 nid=0x4443 in Object.wait() [0x8299b000]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x9505e188> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
       - locked <0x9505e188> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
       at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
       at java.lang.Thread.run(Thread.java:619)
      
      "RMI TCP Accept-1098" daemon prio=10 tid=0x81ac7000 nid=0x4442 runnable [0x829ec000]
       java.lang.Thread.State: RUNNABLE
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
       - locked <0x94e78438> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:453)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
       at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
       at java.lang.Thread.run(Thread.java:619)
      
      "Timer-Log4jService" daemon prio=10 tid=0x8142d400 nid=0x4441 in Object.wait() [0x82d9c000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x94e77e20> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x94e77e20> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "ZipFile Lock Reaper" daemon prio=10 tid=0x82ebb800 nid=0x443f in Object.wait() [0x82dfe000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x946c7a10> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x946c7a10> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "Thread-2" prio=10 tid=0x83113800 nid=0x443e in Object.wait() [0x833b2000]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x942aa6e0> (a java.util.Collections$SynchronizedList)
       at java.lang.Object.wait(Object.java:485)
       at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:205)
       - locked <0x942aa6e0> (a java.util.Collections$SynchronizedList)
       at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:575)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:533)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:510)
       at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
       at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
       at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
       at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
       at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:575)
       at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)
       at java.security.AccessController.doPrivileged(Native Method)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:533)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:510)
       at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
       at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
       at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
       at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
       at org.jboss.osgi.plugins.facade.classloading.OSGiClassLoaderDomain.loadClass(OSGiClassLoaderDomain.java:84)
       at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:801)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:444)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
       - locked <0x957501c0> (a org.jboss.classloader.spi.base.BaseClassLoader)
       at org.jboss.osgi.webconsole.internal.WebConsoleActivator.start(WebConsoleActivator.java:41)
       at org.jboss.osgi.plugins.facade.bundle.OSGiBundleState.startInternal(OSGiBundleState.java:300)
       at org.jboss.osgi.plugins.deployers.bundle.OSGiBundleActivatorDeployer.deploy(OSGiBundleActivatorDeployer.java:49)
       at org.jboss.osgi.plugins.deployers.bundle.OSGiBundleActivatorDeployer.deploy(OSGiBundleActivatorDeployer.java:1)
       at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
       at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
       at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1440)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1158)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1099)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:643)
       at org.jboss.osgi.plugins.deployers.bundle.OSGiDeployersWrapper.change(OSGiDeployersWrapper.java:171)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.change(MainDeployerImpl.java:755)
       at org.jboss.osgi.plugins.deployers.bundle.OSGiDeployersWrapper.afterDeployersProcess(OSGiDeployersWrapper.java:157)
       at org.jboss.osgi.plugins.deployers.bundle.OSGiDeployersWrapper.process(OSGiDeployersWrapper.java:102)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
       at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
       at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
       at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
       at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:775)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
       at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:258)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:97)
       at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:851)
       at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:432)
       - locked <0x945cba98> (a org.jboss.bootstrap.impl.as.server.JBossASServerImpl)
       at java.lang.Thread.run(Thread.java:619)
      
      "DestroyJavaVM" prio=10 tid=0xb7407000 nid=0x4430 waiting on condition [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "main" prio=10 tid=0xb74eac00 nid=0x443d in Object.wait() [0x8349c000]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x945ef6d8> (a java.lang.Thread)
       at java.lang.Thread.join(Thread.java:1143)
       - locked <0x945ef6d8> (a java.lang.Thread)
       at java.lang.Thread.join(Thread.java:1196)
       at org.jboss.bootstrap.impl.base.server.AbstractServer.start(AbstractServer.java:357)
       at org.jboss.bootstrap.impl.mc.server.AbstractMCServerBase.start(AbstractMCServerBase.java:238)
       at org.jboss.Main.boot(Main.java:386)
       at org.jboss.Main$1.run(Main.java:715)
       at java.lang.Thread.run(Thread.java:619)
      
      "Low Memory Detector" daemon prio=10 tid=0xb74b1400 nid=0x443b runnable [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "CompilerThread1" daemon prio=10 tid=0xb74af400 nid=0x443a waiting on condition [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "CompilerThread0" daemon prio=10 tid=0xb74adc00 nid=0x4439 waiting on condition [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "JDWP Event Helper Thread" daemon prio=10 tid=0xb74ac000 nid=0x4438 runnable [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "JDWP Transport Listener: dt_socket" daemon prio=10 tid=0xb74a9c00 nid=0x4437 runnable [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "Signal Dispatcher" daemon prio=10 tid=0xb74a2000 nid=0x4436 waiting on condition [0x00000000]
       java.lang.Thread.State: RUNNABLE
      
      "Finalizer" daemon prio=10 tid=0xb7493000 nid=0x4435 in Object.wait() [0x83b7d000]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x944aa858> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
       - locked <0x944aa858> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0xb7491c00 nid=0x4434 in Object.wait() [0x837b4000]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x944aa810> (a java.lang.ref.Reference$Lock)
       at java.lang.Object.wait(Object.java:485)
       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
       - locked <0x944aa810> (a java.lang.ref.Reference$Lock)
      
      "VM Thread" prio=10 tid=0xb748dc00 nid=0x4433 runnable
      
      "GC task thread#0 (ParallelGC)" prio=10 tid=0xb740e800 nid=0x4431 runnable
      
      "GC task thread#1 (ParallelGC)" prio=10 tid=0xb740fc00 nid=0x4432 runnable
      
      "VM Periodic Task Thread" prio=10 tid=0xb74b3000 nid=0x443c waiting on condition
      
      JNI global references: 4868
      
      Heap
       PSYoungGen total 24896K, used 12027K [0xb07f0000, 0xb2580000, 0xb40d0000)
       eden space 21440K, 56% used [0xb07f0000,0xb13aad78,0xb1ce0000)
       from space 3456K, 0% used [0xb2220000,0xb2224000,0xb2580000)
       to space 4416K, 0% used [0xb1ce0000,0xb1ce0000,0xb2130000)
       PSOldGen total 116544K, used 26148K [0x940d0000, 0x9b2a0000, 0xb07f0000)
       object space 116544K, 22% used [0x940d0000,0x95a59090,0x9b2a0000)
       PSPermGen total 36736K, used 19572K [0x840d0000, 0x864b0000, 0x940d0000)
       object space 36736K, 53% used [0x840d0000,0x853ed3b0,0x864b0000)
      
      


        • 1. Re: Potential deadlock in ClassLoaderManager

          No the problem is on line 205 - unless the first thread with the add() is looping.

          It looks like a missed notification since nothing is holding the lock?

          "Thread-2" prio=10 tid=0x83113800 nid=0x443e in Object.wait() [0x833b2000]
           java.lang.Thread.State: WAITING (on object monitor)
           at java.lang.Object.wait(Native Method)
           - waiting on <0x942aa6e0> (a java.util.Collections$SynchronizedList)
           at java.lang.Object.wait(Object.java:485)
           at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:205)
           - locked <0x942aa6e0> (a java.util.Collections$SynchronizedList)
          


          If you can reproduce it, can you turn up the logging for
          org.jboss.detailed.classloader.ClassLoaderManager
          to TRACE.

          • 2. Re: Potential deadlock in ClassLoaderManager
            thomas.diesler

            Ok, I'll have a look

            • 3. Re: Potential deadlock in ClassLoaderManager
              thomas.diesler

              HttpServiceTestCase approximately fails on every third run (without trace logging). In which case you would see

              [tdiesler@tdvaio example]$ mvn -Dtest=HttpServiceTestCase test
              
              Running org.jboss.test.osgi.example.http.HttpServiceTestCase
              FIXME [JBCL-116] Potential deadlock in ClassLoaderManager
              Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.233
              


              With trace logging enabled the issue occurs less frequent. Nevertheless I managed to catch it and attached the trace to the issue.

               <category name="org.jboss.detailed.classloader">
               <priority value="TRACE" />
               </category>
               <category name="org.jboss.classloader">
               <priority value="TRACE" />
               </category>
              


              https://jira.jboss.org/jira/browse/JBCL-116


              • 4. Re: Potential deadlock in ClassLoaderManager
                wolfc

                 

                at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java
                :128)
                 - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)
                 - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)

                taskList == toTaskList

                • 5. Re: Potential deadlock in ClassLoaderManager

                   

                  "wolfc" wrote:
                  at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java
                  :128)
                   - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)
                   - locked <0x95a17ae8> (a java.util.Collections$SynchronizedList)

                  taskList == toTaskList


                  Yes, that's what I said above about it looping.

                  We had that problem before
                  https://jira.jboss.org/jira/browse/JBCL-81
                  which I fixed by making the code more like the UCL stuff from 4.2.x

                  This maybe a different variation of the problem (one which would also make the
                  4.2.x code loop as well), but I can't tell.

                  I can't reproduce the problem (although Thomas says he can) and the TRACE
                  logging he posted on the JIRA issue doesn't match the thread dump.
                  There is no "Reassigning task:" logging at all in his log which we would expect
                  from stacktrace since it occurs just before the list.add()

                  There are some other incongruencies with the log as well
                  (e.g. the thread dump shows it using a BaseClassLoader as the parent
                  classloader for the Domain, but the log shows all the parents to be the JDK classloader).

                  Anyway, I've applied a fix that stops this add() looping if it tries to reassign
                  ThreadTasks to itself. This is in a snapshot here (number 14):
                  http://snapshots.jboss.org/maven2/org/jboss/cl/jboss-classloader/2.0.7-SNAPSHOT/
                  So we can see whether this fixes the problem Thomas is seeing.