1 2 Previous Next 29 Replies Latest reply on Feb 4, 2009 9:39 AM by Adrian Brock

    Strange classloading behavior -- thread stuck

    Brian Stansberry Master

      When testing mod_cluster, Bela's reported a couple different occurences where a thread appears to be unable to make progress inside some jboss-cl code. The thread is bringing messages up from the cluster which has the effect of locking up the cluster.

      Bela provided some thread dump info, which I'll post below. I've been poking around in the relevant code, but it's intricate enough that I'd better call in the experts. In all cases the thread that isn't progressing is inside this section of ClassLoaderManager.unregisterLoaderInThread():

      // Any ThreadTasks associated with this thread must be reassigned
       List<ThreadTask> taskList = loadTasksByThread.get(thread);
       if (taskList != null)
       {
       synchronized (taskList)
       {
       while (taskList.isEmpty() == false)
       {
       ThreadTask threadTask = taskList.remove(0);
       ClassLoadingTask loadTask = threadTask.getLoadTask();
       Thread requestingThread = loadTask.getRequestingThread();
       if( trace )
       log.trace("Reassigning task: " + threadTask+" to " + requestingThread);
       threadTask.setThread(null);
       // Insert the task into the front of requestingThread task list
       List<ThreadTask> toTaskList = loadTasksByThread.get(requestingThread);
       synchronized (toTaskList)
       {
       toTaskList.add(0, threadTask);
       loadTask.nextEvent();
       toTaskList.notify();
       }
       }
       }
       }


      Specifically, it's inside the "while (taskList.isEmpty() == false)" block, which leads me to question whether somehow the taskList is never empty. (Uneducated guess: somehow taskList == toTaskList, so this becomes a loop adding and removing threadTask???)

      Will post stack traces separately and inform Bela of this thread so he can answer any questions about what exactly he was doing.

        • 1. Re: Strange classloading behavior -- thread stuck
          Brian Stansberry Master

          First set of thread dumps. Following are two dumps showing the same thread. Following is some back and forth between us about when this happened:

          "Bela Ban" wrote:
          "Brian Stansberry" wrote:
          A few questions:

          1) Is this with a stock AS 5.0.0.GA and a standard install of mod_cluster? I think yes, but am double-checking?


          Yes


          "Brian Stansberry" wrote:
          2) Is this reproducible?


          Yes. If you run https://jira.jboss.org/jira/browse/MODCLUSTER-31 (IIRC), you always have these long stack traces.


          First stack trace for the problem thread:


          Name: Incoming-12,192.168.1.5:57544
          State: RUNNABLE
          Total blocked: 4 Total waited: 24,105
          
          Stack trace:
          java.util.Collections$SynchronizedList.remove(Collections.java:1825)
           - locked java.util.Collections$SynchronizedList@45b84e09
          org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:115)
           - locked java.util.Collections$SynchronizedList@45b84e09
          org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
           - locked org.jboss.classloader.spi.base.BaseClassLoader@7b17ad7b
          org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
          org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
          org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
          org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
          org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
          org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
          org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
           - locked org.jboss.classloader.spi.base.BaseClassLoader@3bbe9a50
          java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
           - locked org.jboss.classloader.spi.base.BaseClassLoader@3bbe9a50
          java.lang.ClassLoader.defineClass1(Native Method)
          java.lang.ClassLoader.defineClass(ClassLoader.java:621)
          org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
          org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
          org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
          java.security.AccessController.doPrivileged(Native Method)
          org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
           - locked org.jboss.classloader.spi.base.BaseClassLoader@3bbe9a50
          org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
          org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
          org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
          org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
          org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
          org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
          org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
          org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
          org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
          org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
           - locked org.jboss.classloader.spi.base.BaseClassLoader@3bbe9a50
          java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
          java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
          java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
          java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
          java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
          java.util.HashSet.readObject(HashSet.java:291)
          sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          java.lang.reflect.Method.invoke(Method.java:597)
          java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
          java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
          java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
          java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
          java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
          java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
          java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
          org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferResponseInternal(ClusterPartition.java:1887)
          org.jboss.ha.framework.server.ClusterPartition$ResponseMarshallerImpl.objectFromByteBuffer(ClusterPartition.java:2084)
          org.jgroups.blocks.RpcDispatcher$MarshallerAdapter.objectFromByteBuffer(RpcDispatcher.java:138)
          org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:557)
          org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
          org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
          org.jgroups.JChannel.up(JChannel.java:1274)
          org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
          org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:377)
          org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
          org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
          org.jgroups.protocols.FC.up(FC.java:468)
          org.jgroups.protocols.pbcast.GMS.up(GMS.java:796)
          org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
          org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
          org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
          org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:747)
          org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
          org.jgroups.protocols.FD.up(FD.java:284)
          org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
          org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
          org.jgroups.protocols.Discovery.up(Discovery.java:263)
          org.jgroups.protocols.PING.up(PING.java:270)
          org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
          org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
          org.jgroups.protocols.TP.access$100(TP.java:49)
          org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
          org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          java.lang.Thread.run(Thread.java:619)
          


          And a second one captured later:

          "Incoming-12,192.168.1.5:57544" prio=10 tid=0x00000000455b1000 nid=0x7b55 runnable [0x000000003ed7f000..0x000000003ed82bf0]
           java.lang.Thread.State: RUNNABLE
           at java.util.Collections$SynchronizedCollection.isEmpty(Collections.java:1560)
           - locked <0x000000001d3ce1d0> (a java.util.Collections$SynchronizedList)
           at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:113)
           - locked <0x000000001d3ce1d0> (a java.util.Collections$SynchronizedList)
           at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
           - locked <0x000000001ac6f480> (a org.jboss.classloader.spi.base.BaseClassLoader)
           at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
           at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
           at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
           at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
           at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
           - locked <0x0000000015c73390> (a org.jboss.classloader.spi.base.BaseClassLoader)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
           at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
           - locked <0x0000000015c73390> (a org.jboss.classloader.spi.base.BaseClassLoader)
           at java.lang.ClassLoader.defineClass1(Native Method)
           at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
           at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
           at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
           at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
           at java.security.AccessController.doPrivileged(Native Method)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
           - locked <0x0000000015c73390> (a org.jboss.classloader.spi.base.BaseClassLoader)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
           at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
           at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
           at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
           at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
           at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
           at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
           at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
           at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
           - locked <0x0000000015c73390> (a org.jboss.classloader.spi.base.BaseClassLoader)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
           at org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
           at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
           at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
           at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
           at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
           at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
           at java.util.HashSet.readObject(HashSet.java:291)
           at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
           at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
           at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
           at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
           at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
           at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
           at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
           at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
           at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
           at org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferResponseInternal(ClusterPartition.java:1887)
           at org.jboss.ha.framework.server.ClusterPartition$ResponseMarshallerImpl.objectFromByteBuffer(ClusterPartition.java:2084)
           at org.jgroups.blocks.RpcDispatcher$MarshallerAdapter.objectFromByteBuffer(RpcDispatcher.java:138)
           at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:557)
           at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
           at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
           at org.jgroups.JChannel.up(JChannel.java:1274)
           at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
           at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:377)
           at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
           at org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
           at org.jgroups.protocols.FC.up(FC.java:468)
           at org.jgroups.protocols.pbcast.GMS.up(GMS.java:796)
           at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
           at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
           at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
           at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:747)
           at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
           at org.jgroups.protocols.FD.up(FD.java:284)
           at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
           at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
           at org.jgroups.protocols.Discovery.up(Discovery.java:263)
           at org.jgroups.protocols.PING.up(PING.java:270)
           at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
           at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
           at org.jgroups.protocols.TP.access$100(TP.java:49)
           at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
           at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
           at java.lang.Thread.run(Thread.java:619)
          


          • 2. Re: Strange classloading behavior -- thread stuck
            Brian Stansberry Master

            Second set of stack traces taken today w/ mod_cluster 1.0.0.Beta3; not sure what AS version but I expect still 5.0.0.GA.

            First stack trace for problem thread:

            "Incoming-2,192.168.1.5:42274" prio=10 tid=0x0000000040353800 nid=0x2b3a runnable [0x000000003a2fc000..0x000000003a2ffb70]
             java.lang.Thread.State: RUNNABLE
             at java.lang.Object.notify(Native Method)
             at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:127)
             - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
             - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
             at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
             - locked <0x00000000199bace8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
             at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
             at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
             at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.defineClass1(Native Method)
             at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
             at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
             at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
             at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
             at java.security.AccessController.doPrivileged(Native Method)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
             at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
             at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
             at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
             at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
             at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
             at org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
             at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
             at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
             at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
             at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
             at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
             at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
             at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
             at org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferResponseInternal(ClusterPartition.java:1887)
             at org.jboss.ha.framework.server.ClusterPartition$ResponseMarshallerImpl.objectFromByteBuffer(ClusterPartition.java:2084)
             at org.jgroups.blocks.RpcDispatcher$MarshallerAdapter.objectFromByteBuffer(RpcDispatcher.java:138)
             at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:557)
             at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
             at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
             at org.jgroups.JChannel.up(JChannel.java:1274)
             at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
             at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:377)
             at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
             at org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
             at org.jgroups.protocols.FC.up(FC.java:468)
             at org.jgroups.protocols.pbcast.GMS.up(GMS.java:796)
             at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
             at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
             at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
             at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:747)
             at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
             at org.jgroups.protocols.FD.up(FD.java:284)
             at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
             at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
             at org.jgroups.protocols.Discovery.up(Discovery.java:263)
             at org.jgroups.protocols.PING.up(PING.java:270)
             at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
             at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
             at org.jgroups.protocols.TP.access$100(TP.java:49)
             at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
             at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
             at
             java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
             at java.lang.Thread.run(Thread.java:619)
            


            Another thread Bela included in his problem report; not sure if/how it relates and whether it the trace was taken at same time as the one above or later, with the second version below.

            "ajp-192.168.1.5-8009-2" daemon prio=10 tid=0x000000004575b800 nid=0x2b76 in Object.wait() [0x00000000028b6000..0x00000000028b9c70]
             java.lang.Thread.State: WAITING (on object monitor)
             at java.lang.Object.wait(Native Method)
             - waiting on <0x000000001dbed570> (a java.util.Collections$SynchronizedList)
             at java.lang.Object.wait(Object.java:485)
             at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:207)
             - locked <0x000000001dbed570> (a java.util.Collections$SynchronizedList)
             at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
             - locked <0x0000000019928de8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
             at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
             - locked <0x0000000019928de8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at org.apache.jasper.compiler.JspUtil.<clinit>(JspUtil.java:66)
             at org.apache.jasper.JspCompilationContext.getServletClassName(JspCompilationContext.java:370)
             at org.apache.jasper.JspCompilationContext.getClassFileName(JspCompilationContext.java:510)
             at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:421)
             at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:369)
             at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:581)
             at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:312)
             - locked <0x000000001dbed890> (a org.apache.jasper.servlet.JspServletWrapper)
             at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:322)
             at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:249)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
             at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
             at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
             at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
             at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
             at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
             at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
             at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
             at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
             at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:109)
             at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:87)
             at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
             at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
             at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
             at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
             at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
             at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
             at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
             at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
             at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
             at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
             at java.lang.Thread.run(Thread.java:619)


            A second stack trace taken later for the JGroups "Incoming" thread shown above:

            "Incoming-2,192.168.1.5:42274" prio=10 tid=0x0000000040353800 nid=0x2b3a runnable [0x000000003a2fc000..0x000000003a2ffb70]
             java.lang.Thread.State: RUNNABLE
             at java.lang.Object.notify(Native Method)
             at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:127)
             - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
             - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
             at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
             - locked <0x00000000199bace8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
             at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
             at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
             at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.defineClass1(Native Method)
             at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
             at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
             at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
             at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
             at java.security.AccessController.doPrivileged(Native Method)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
             at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
             at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
             at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
             at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
             at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
             at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
             at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
             - locked <0x000000001509f8e8> (a org.jboss.classloader.spi.base.BaseClassLoader)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
             at org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
             at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
             at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
             at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
             at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
             at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
             at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
             at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
             at org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferResponseInternal(ClusterPartition.java:1887)
             at org.jboss.ha.framework.server.ClusterPartition$ResponseMarshallerImpl.objectFromByteBuffer(ClusterPartition.java:2084)
             at org.jgroups.blocks.RpcDispatcher$MarshallerAdapter.objectFromByteBuffer(RpcDispatcher.java:138)
             at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:557)
             at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
             at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
             at org.jgroups.JChannel.up(JChannel.java:1274)
             at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
             at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:377)
             at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
             at org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
             at org.jgroups.protocols.FC.up(FC.java:468)
             at org.jgroups.protocols.pbcast.GMS.up(GMS.java:796)
             at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
             at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
             at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
             at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:747)
             at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
             at org.jgroups.protocols.FD.up(FD.java:284)
             at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
             at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
             at org.jgroups.protocols.Discovery.up(Discovery.java:263)
             at org.jgroups.protocols.PING.up(PING.java:270)
             at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
             at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
             at org.jgroups.protocols.TP.access$100(TP.java:49)
             at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
             at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
             at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
             at java.lang.Thread.run(Thread.java:619)
            


            Here the "Incoming" thread is caught at the same point in both dumps. Unlikely it's somehow stuck in java.lang.Object.notify() so seems likely to me that the thread's in a tight loop.

            • 3. Re: Strange classloading behavior -- thread stuck
              Brian Stansberry Master

              Probably unrelated but when poking around trying to understand this code I noticed this in ClassLoaderManager.nextTask()

               synchronized (taskList)
               {
               ....
              
               // See if the task is complete
               if (task.getThreadTaskCount() == 0)
               {
               task.finish();
               log.trace("End nextTask(FINISHED), task="+task);
               return;
               }
               }
              
               ThreadTask threadTask = taskList.remove(0);
               ClassLoadingTask loadTask = threadTask.getLoadTask();


              The taskList is unguarded during the remove(0) call since the call is outside the synchronized block.

              • 4. Re: Strange classloading behavior -- thread stuck
                Adrian Brock Master

                 

                "bstansberry@jboss.com" wrote:

                Specifically, it's inside the "while (taskList.isEmpty() == false)" block, which leads me to question whether somehow the taskList is never empty. (Uneducated guess: somehow taskList == toTaskList, so this becomes a loop adding and removing threadTask???)


                I'm also not totally sure how this code works either. :-)
                It is a copy of LoadMgr3 from the UnifiedClassLoader which I tried to simplify
                but couldn't get it work so I left it as it was. ;-)

                But I don't believe taskList == toTaskList should ever be true.

                The purpose of this loop is to reassign classloading tasks to other threads because
                this thread has finished doing its classloading. It shouldn't be the "requestingThread"
                on those classloading tasks. If it were, the nextTask() should have already done them.

                Simplifying your second set of stack traces:
                Trace1:
                "Incoming-2,192.168.1.5:42274" prio=10 tid=0x0000000040353800 nid=0x2b3a runnable [0x000000003a2fc000..0x000000003a2ffb70]
                 java.lang.Thread.State: RUNNABLE
                 at java.lang.Object.notify(Native Method)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManag
                er.java:127)
                 - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
                 - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
                 at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                 - locked <0x00000000199bace8> (a org.jboss.classloader.spi.base.BaseClassLoader)
                 at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                
                Trace 2:
                "ajp-192.168.1.5-8009-2" daemon prio=10 tid=0x000000004575b800 nid=0x2b76 in Object.wait() [0x00000000028b6000..0x0000000002
                8b9c70]
                 java.lang.Thread.State: WAITING (on object monitor)
                 at java.lang.Object.wait(Native Method)
                 - waiting on <0x000000001dbed570> (a java.util.Collections$SynchronizedList)
                 at java.lang.Object.wait(Object.java:485)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:207)
                 - locked <0x000000001dbed570> (a java.util.Collections$SynchronizedList)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                
                Trace 3:
                "Incoming-2,192.168.1.5:42274" prio=10 tid=0x0000000040353800 nid=0x2b3a runnable [0x000000003a2fc000..0x000000003a2ffb70]
                 java.lang.Thread.State: RUNNABLE
                 at java.lang.Object.notify(Native Method)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManag
                er.java:127)
                 - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
                 - locked <0x000000001ce12608> (a java.util.Collections$SynchronizedList)
                 at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                 - locked <0x00000000199bace8> (a org.jboss.classloader.spi.base.BaseClassLoader)
                 at org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                 at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                


                It's hard to tell without TRACE logging being enabled, but if Trace3 is really showing
                that thread still looping from Trace1 then it looks your "uneducated guess" is correct.

                i.e. Trace2 is showing a thread for the same classloader waiting on its thread task list
                (0x000000001dbed570) while Trace1 and Trace3 is showing two locks on
                (0x000000001ce12608).

                 synchronized (taskList)
                 {
                 while (taskList.isEmpty() == false) // FIRST LOCK
                 {
                 ThreadTask threadTask = taskList.remove(0);
                 ClassLoadingTask loadTask = threadTask.getLoadTask();
                 Thread requestingThread = loadTask.getRequestingThread();
                 if( trace )
                 log.trace("Reassigning task: " + threadTask+" to " + requestingThread);
                 threadTask.setThread(null);
                 // Insert the task into the front of requestingThread task list
                 List<ThreadTask> toTaskList = loadTasksByThread.get(requestingThread);
                 synchronized (toTaskList) // SECOND LOCK
                 {
                 toTaskList.add(0, threadTask);
                 loadTask.nextEvent();
                 toTaskList.notify();
                 }
                 }
                 }
                


                So the issue is to try to figure out how that could happen.

                If you know how to reproduce the problem, enabling TRACE logging
                for org.jboss.classloader would help greatly. ;-)

                I don't think your second guess is correct? The one about the remove
                being outside the synchronized block. The code is the same as JBoss4.x

                • 5. Re: Strange classloading behavior -- thread stuck
                  Adrian Brock Master

                   

                  "adrian@jboss.org" wrote:

                  The code is the same as JBoss4.x


                  There is one difference between 4.x and 5.x, which is the 4.x code won't loop forever
                  since it uses a for loop
                  JBoss 4.x
                  
                   int size = taskList != null ? taskList.size() : 0;
                   synchronized( taskList )
                   {
                   for(int i = 0; i < size; i ++)
                  
                  JBoss5.x
                  
                   synchronized (taskList)
                   {
                   while (taskList.isEmpty() == false)
                   {
                  


                  But using a for loop would just hide the problem.

                  The thread on Trace2 still wouldn't be woken up because the other thread wouldn't
                  do the notification on its task list (it would still assign the task to the wrong thread).

                  • 6. Re: Strange classloading behavior -- thread stuck
                    Ales Justin Master

                    Thread news jump-in ...

                    Whatever you two come up with
                    make it fast if you wanna see this in 5.0.1. :-)

                    As I'll probably do full MC release tomorrow,
                    now that VFS is in semi decent shape. ;-)

                    • 7. Re: Strange classloading behavior -- thread stuck
                      Brian Stansberry Master

                       

                      "adrian@jboss.org" wrote:

                      So the issue is to try to figure out how that could happen.

                      If you know how to reproduce the problem, enabling TRACE logging
                      for org.jboss.classloader would help greatly. ;-)


                      Bela seems able to reproduce easily enough; I've asked him to add TRACE logging.

                      He sent me another message this AM with a bunch of stack traces from the problem thread. All showed the thread in different places inside the while (taskList.isEmpty() == false) loop.


                      I don't think your second guess is correct? The one about the remove
                      being outside the synchronized block. The code is the same as JBoss4.x


                      That was more something I noticed that just looked wrong in general -- mutating an unguarded object. If it caused a problem though I'd expect it to be an IndexOutOfBoundsException because a thread lost a race to remove the last item.

                      • 8. Re: Strange classloading behavior -- thread stuck
                        Bela Ban Master

                        Howdy, folks, I concur with Brian's assertion that the offending code seems to be looping. I'll enable TRACE for the org.jboss.classloader package, hopefully today, and post the findings. I'll past 4 (partial) stack traces of the offending Incoming-2 thread, each taken a few seconds apart, below. You can see that the code seems to loop indeed:

                        Name: Incoming-2,192.168.1.9:49336
                        State: RUNNABLE
                        Total blocked: 22 Total waited: 1,719

                        Stack trace:
                        java.util.Collections$SynchronizedList.add(Collections.java:1822)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:125)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@783cdf2d
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.defineClass1(Native Method)
                        java.lang.ClassLoader.defineClass(ClassLoader.java:621)
                        org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
                        java.security.AccessController.doPrivileged(Native Method)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
                        org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                        org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                        org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
                        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
                        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        java.util.HashSet.readObject(HashSet.java:291)
                        sun.reflect.GeneratedMethodAccessor219.invoke(Unknown Source)
                        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        java.lang.reflect.Method.invoke(Method.java:597)
                        java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
                        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        org.jgroups.blocks.MethodCall.readExternal(MethodCall.java:488)
                        java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferInternal(ClusterPartition.java:1852)
                        org.jboss.ha.framework.server.ClusterPartition$RpcHandler.handle(ClusterPartition.java:2218)
                        org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
                        org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
                        org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
                        org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
                        org.jgroups.JChannel.up(JChannel.java:1274)
                        org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
                        org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:443)
                        org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
                        org.jgroups.protocols.FRAG2.up(FRAG2.java:192)
                        org.jgroups.protocols.FC.up(FC.java:468)
                        org.jgroups.protocols.pbcast.GMS.up(GMS.java:796)
                        org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
                        org.jgroups.protocols.UNICAST.up(UNICAST.java:299)
                        org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:873)
                        org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:705)
                        org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
                        org.jgroups.protocols.FD.up(FD.java:284)
                        org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
                        org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
                        org.jgroups.protocols.Discovery.up(Discovery.java:263)
                        org.jgroups.protocols.PING.up(PING.java:270)
                        org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
                        org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
                        org.jgroups.protocols.TP.access$100(TP.java:49)
                        org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
                        org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
                        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                        java.lang.Thread.run(Thread.java:619)




                        Name: Incoming-2,192.168.1.9:49336
                        State: RUNNABLE
                        Total blocked: 22 Total waited: 1,719

                        Stack trace:
                        org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:123)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@783cdf2d
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.defineClass1(Native Method)
                        java.lang.ClassLoader.defineClass(ClassLoader.java:621)
                        org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
                        java.security.AccessController.doPrivileged(Native Method)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
                        org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                        org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                        org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
                        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
                        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        java.util.HashSet.readObject(HashSet.java:291)



                        Stack trace:
                        java.util.Collections$SynchronizedList.add(Collections.java:1822)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:125)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@783cdf2d
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.defineClass1(Native Method)
                        java.lang.ClassLoader.defineClass(ClassLoader.java:621)
                        org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
                        java.security.AccessController.doPrivileged(Native Method)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
                        org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                        org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                        org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
                        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
                        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        java.util.HashSet.readObject(HashSet.java:291)
                        sun.reflect.GeneratedMethodAccessor219.invoke(Unknown Source)
                        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        java.lang.reflect.Method.invoke(Method.java:597)
                        java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
                        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        org.jgroups.blocks.MethodCall.readExternal(MethodCall.java:488)
                        java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)


                        Name: Incoming-2,192.168.1.9:49336
                        State: RUNNABLE
                        Total blocked: 22 Total waited: 1,719

                        Stack trace:
                        org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:115)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@783cdf2d
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.defineClass1(Native Method)
                        java.lang.ClassLoader.defineClass(ClassLoader.java:621)
                        org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
                        java.security.AccessController.doPrivileged(Native Method)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
                        org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                        org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                        org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
                        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
                        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        java.util.HashSet.readObject(HashSet.java:291)
                        sun.reflect.GeneratedMethodAccessor219.invoke(Unknown Source)
                        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        java.lang.reflect.Method.invoke(Method.java:597)
                        java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
                        java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        org.jgroups.blocks.MethodCall.readExternal(MethodCall.java:488)
                        java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        org.jboss.ha.framework.server.ClusterPartition.objectFromByteBufferInternal(ClusterPartition.java:1852)


                        Name: Incoming-2,192.168.1.9:49336
                        State: RUNNABLE
                        Total blocked: 22 Total waited: 1,719

                        Stack trace:
                        java.util.Collections$SynchronizedList.remove(Collections.java:1825)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.ClassLoaderManager.unregisterLoaderThread(ClassLoaderManager.java:115)
                        - locked java.util.Collections$SynchronizedList@28ad5127
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:1011)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@783cdf2d
                        org.jboss.classloader.spi.base.BaseClassLoader.unlock(BaseClassLoader.java:894)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:289)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.defineClass1(Native Method)
                        java.lang.ClassLoader.defineClass(ClassLoader.java:621)
                        org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:509)
                        org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:469)
                        java.security.AccessController.doPrivileged(Native Method)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:467)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:444)
                        org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                        org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                        org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                        org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
                        org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
                        org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:735)
                        org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:378)
                        - locked org.jboss.classloader.spi.base.BaseClassLoader@5f14a3c6
                        java.lang.ClassLoader.loadClass(ClassLoader.java:252)
                        org.jboss.invocation.MarshalledValueInputStream.resolveClass(MarshalledValueInputStream.java:101)
                        java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
                        java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
                        java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
                        java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
                        java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
                        java.util.HashSet.readObject(HashSet.java:291)
                        sun.reflect.GeneratedMethodAccessor219.invoke(Unknown Source)
                        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        java.lang.reflect.Method.invoke(Method.java:597)

                        • 9. Re: Strange classloading behavior -- thread stuck
                          Adrian Brock Master

                           

                          "bstansberry@jboss.com" wrote:
                          "adrian@jboss.org" wrote:

                          I don't think your second guess is correct? The one about the remove
                          being outside the synchronized block. The code is the same as JBoss4.x


                          That was more something I noticed that just looked wrong in general -- mutating an unguarded object. If it caused a problem though I'd expect it to be an IndexOutOfBoundsException because a thread lost a race to remove the last item.


                          I don't think there is a problem there (not just because it is the same as JBoss4 :-).

                          The list is a SynchronizedList and you only get to that code if you own the lock
                          and there is something to execute (notice the return inside the synchronized block
                          if the list is empty).

                          So other people can add to the list, but you can be sure there is at least one element
                          in it when you do the remove().

                          • 10. Re: Strange classloading behavior -- thread stuck
                            Adrian Brock Master

                            But where I do see a problem is in ClassLoadingTask.getThreadTaskCount().

                            This is incremented and decremented as ThreadTasks within the
                            overall classloading request (the ClassLoadingTask) are done (possibly by other threads).

                            Those increments and decrements are in synchronized blocks but the getter is not
                            synchronized.

                            So the ClassLoaderManager.process() and ClassLoaderManager.nextTask()
                            could prematurely leave their loops if it is seeing a stale value.

                            I'm not so confident that this is the real problem since I would expect the stale value
                            to go either way and hence see problems with it not leaving the loop if that was the real issue.

                            • 11. Re: Strange classloading behavior -- thread stuck
                              Adrian Brock Master

                               

                              "bela@jboss.com" wrote:
                              Howdy, folks, I concur with Brian's assertion that the offending code seems to be looping. I'll enable TRACE for the org.jboss.classloader package, hopefully today, and post the findings. I'll past 4 (partial) stack traces of the offending Incoming-2 thread, each taken a few seconds apart, below. You can see that the code seems to loop indeed:


                              Thanks Bela. These stacktraces show the same thing, i.e. double locking on the task list
                              which indicates a thread trying to reschedule tasks to itself.

                              • 12. Re: Strange classloading behavior -- thread stuck
                                Adrian Brock Master

                                 

                                "alesj" wrote:
                                Thread news jump-in ...

                                Whatever you two come up with
                                make it fast if you wanna see this in 5.0.1. :-)

                                As I'll probably do full MC release tomorrow,
                                now that VFS is in semi decent shape. ;-)


                                Has 2.0.2.GA been released?
                                I see it in repository.jboss.com but JIRA says it is not.
                                Maven thinks we are on 2.0.3-SNAPSHOT
                                I've raised https://jira.jboss.org/jira/browse/JBCL-81 as fix for 2.0.3.GA

                                • 13. Re: Strange classloading behavior -- thread stuck
                                  Adrian Brock Master

                                   

                                  "adrian@jboss.org" wrote:
                                  But where I do see a problem is in ClassLoadingTask.getThreadTaskCount().


                                  Bela I've uploaded a 2.0.3-SNAPSHOT with this fixed if you want to try it.
                                  You need to replace the jboss-classloader.jar in /lib
                                  http://snapshots.jboss.org/maven2/org/jboss/cl/jboss-classloader/2.0.3-SNAPSHOT/

                                  • 14. Re: Strange classloading behavior -- thread stuck
                                    Ales Justin Master

                                     

                                    "adrian@jboss.org" wrote:

                                    Has 2.0.2.GA been released?
                                    I see it in repository.jboss.com but JIRA says it is not.
                                    Maven thinks we are on 2.0.3-SNAPSHOT
                                    I've raised https://jira.jboss.org/jira/browse/JBCL-81 as fix for 2.0.3.GA

                                    Yes, it's been released,
                                    just not properly JIRA-ized due to 5.0.1 catch-up.

                                    I'll do it properly once I'm done testing the 5_0, 5_x branches
                                    with my latest MC snapshots.

                                    If you get the fix in till tomorrow 23:59 CET,
                                    I'll do a full MC release on Thursday, including CL and this fix.


                                    1 2 Previous Next