5 Replies Latest reply on Jul 22, 2008 7:19 PM by theoverlord

    JBM thread leak?

    theoverlord

      I'm using JBM 1.4.0.SP3_CP03 to send messages.

      The messages are produced on clustered JBoss AS from a stateless session bean which uses a clustered queue residing on the clustered servers.

      The destination for the messaged are remote MDBs on external non-clustered JBoss AS.

      The MDB reply with a response using a replyTo queue pattern.

      After sending many messages, I'm unable to send any messages at all - it all gets stuck and the only way I know how to get it working again is to restart the servers.

      I was looking at the JMX console on the clustered servers and I noticed all these threads that look like this:

      Thread: jbm-server-session-76-yg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:391, threadState:WAITING, lockName:java.lang.Object@931c53
      
       java.lang.Object.wait(Native Method)
       java.lang.Object.wait(Object.java:474)
       EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
       EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
       java.lang.Thread.run(Thread.java:595)
      


      There was no less than 100 of these threads shown in the JMX console when it all got stuck.

      Does that look normal?

      It looks highly suspicious to me.

      Why is it creating so many of these threads?

        • 1. Re: JBM thread leak?
          theoverlord

          Here's the full list of JBM related threads found in the JMX console:

          Thread: jbm-cf-jndimapper : priority:5, demon:false, threadId:104, threadState:WAITING, lockName:java.lang.Object@1ceedff
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-reply-executor : priority:5, demon:false, threadId:107, threadState:WAITING, lockName:java.lang.Object@6b2b55
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-response-executor : priority:5, demon:false, threadId:108, threadState:WAITING, lockName:java.lang.Object@1e9f55c
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-e-i26vtxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:147, threadState:WAITING, lockName:java.lang.Object@eb86ab
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-client-session-e-i26vtxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:148, threadState:WAITING, lockName:java.lang.Object@d64e41
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-i-sm7vtxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:153, threadState:WAITING, lockName:java.lang.Object@1aba04b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-client-session-r-5i9vtxif-1-cbnutxif-uj16qe-s36hs4 : priority:5, demon:true, threadId:159, threadState:WAITING, lockName:java.lang.Object@a8d63a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-k2-jbqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:278, threadState:WAITING, lockName:java.lang.Object@8897ec
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-l2-cyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:279, threadState:WAITING, lockName:java.lang.Object@1ae18aa
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-m2-dyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:280, threadState:WAITING, lockName:java.lang.Object@d55c6e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-n2-fyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:281, threadState:WAITING, lockName:java.lang.Object@1a756e8
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-o2-hyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:282, threadState:WAITING, lockName:java.lang.Object@13ab9fe
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-p2-iyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:283, threadState:WAITING, lockName:java.lang.Object@88d45a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-q2-kyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:284, threadState:WAITING, lockName:java.lang.Object@12d6309
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-r2-myqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:285, threadState:WAITING, lockName:java.lang.Object@c0fe1e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-s2-nyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:286, threadState:WAITING, lockName:java.lang.Object@6b994c
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-t2-qyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:287, threadState:WAITING, lockName:java.lang.Object@1d78e7
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-u2-ryqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:288, threadState:WAITING, lockName:java.lang.Object@ec9f65
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-v2-tyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:289, threadState:WAITING, lockName:java.lang.Object@d4b6ed
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-w2-uyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:290, threadState:WAITING, lockName:java.lang.Object@135b44b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-x2-wyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:291, threadState:WAITING, lockName:java.lang.Object@cfe71f
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-y2-xyqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:292, threadState:WAITING, lockName:java.lang.Object@14ea5aa
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-z2-0zqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:293, threadState:WAITING, lockName:java.lang.Object@5e4bf4
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-03-5zqytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:294, threadState:WAITING, lockName:java.lang.Object@ccb3a6
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-83-q6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:295, threadState:WAITING, lockName:java.lang.Object@1268e11
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-93-s6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:296, threadState:WAITING, lockName:java.lang.Object@391125
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-a3-u6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:297, threadState:WAITING, lockName:java.lang.Object@949754
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-b3-v6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:298, threadState:WAITING, lockName:java.lang.Object@15b123f
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-c3-x6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:299, threadState:WAITING, lockName:java.lang.Object@b1f36
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-d3-z6rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:300, threadState:WAITING, lockName:java.lang.Object@2e3520
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-e3-07rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:301, threadState:WAITING, lockName:java.lang.Object@1a1e3aa
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-f3-27rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:302, threadState:WAITING, lockName:java.lang.Object@169b7f7
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-g3-47rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:303, threadState:WAITING, lockName:java.lang.Object@1acef32
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-h3-57rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:304, threadState:WAITING, lockName:java.lang.Object@8bc9cf
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-i3-77rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:305, threadState:WAITING, lockName:java.lang.Object@16e71d6
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-j3-87rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:306, threadState:WAITING, lockName:java.lang.Object@1f3c364
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-k3-a7rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:307, threadState:WAITING, lockName:java.lang.Object@a117c2
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-l3-d7rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:308, threadState:WAITING, lockName:java.lang.Object@1ed457
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-m3-f7rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:309, threadState:WAITING, lockName:java.lang.Object@6c2cb
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-n3-h7rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:310, threadState:WAITING, lockName:java.lang.Object@1dedd0a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-o3-i7rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:311, threadState:WAITING, lockName:java.lang.Object@81cbde
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-w3-i9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:312, threadState:WAITING, lockName:java.lang.Object@16f8d51
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-x3-k9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:313, threadState:WAITING, lockName:java.lang.Object@a6f6eb
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-y3-l9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:314, threadState:WAITING, lockName:java.lang.Object@19ccbf8
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-z3-n9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:315, threadState:WAITING, lockName:java.lang.Object@1132b83
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-04-p9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:316, threadState:WAITING, lockName:java.lang.Object@18da915
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-14-r9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:317, threadState:WAITING, lockName:java.lang.Object@157e112
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-24-s9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:318, threadState:WAITING, lockName:java.lang.Object@796da3
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-34-u9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:319, threadState:WAITING, lockName:java.lang.Object@cf8fd
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-44-w9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:320, threadState:WAITING, lockName:java.lang.Object@1aeac1e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-54-x9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:321, threadState:WAITING, lockName:java.lang.Object@a61eda
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-64-z9rytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:322, threadState:WAITING, lockName:java.lang.Object@339537
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-74-0arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:323, threadState:WAITING, lockName:java.lang.Object@89510f
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-84-2arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:324, threadState:WAITING, lockName:java.lang.Object@128c286
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-94-3arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:325, threadState:WAITING, lockName:java.lang.Object@30342a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-a4-5arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:326, threadState:WAITING, lockName:java.lang.Object@b0e6d6
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-b4-6arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:327, threadState:WAITING, lockName:java.lang.Object@306a0
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-c4-8arytxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:328, threadState:WAITING, lockName:java.lang.Object@a81856
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-q4-a540uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:350, threadState:WAITING, lockName:java.lang.Object@1d5bc97
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-r4-ma80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:353, threadState:WAITING, lockName:java.lang.Object@15096dd
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-s4-oa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:354, threadState:WAITING, lockName:java.lang.Object@1e0a490
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-t4-oa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:355, threadState:WAITING, lockName:java.lang.Object@1541809
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-u4-qa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:356, threadState:WAITING, lockName:java.lang.Object@1f60d18
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-v4-ra80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:357, threadState:WAITING, lockName:java.lang.Object@d5b206
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-w4-sa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:358, threadState:WAITING, lockName:java.lang.Object@19aaa64
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-x4-sa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:359, threadState:WAITING, lockName:java.lang.Object@128c166
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-y4-ta80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:360, threadState:WAITING, lockName:java.lang.Object@b124a6
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-z4-ua80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:361, threadState:WAITING, lockName:java.lang.Object@1dd284a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-05-va80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:362, threadState:WAITING, lockName:java.lang.Object@80185b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-15-wa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:363, threadState:WAITING, lockName:java.lang.Object@1bf31e9
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-25-wa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:364, threadState:WAITING, lockName:java.lang.Object@165c326
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-35-xa80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:365, threadState:WAITING, lockName:java.lang.Object@1ef1241
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-45-ya80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:366, threadState:WAITING, lockName:java.lang.Object@9774a7
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-55-za80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:367, threadState:WAITING, lockName:java.lang.Object@16b8145
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-65-1b80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:368, threadState:WAITING, lockName:java.lang.Object@f3ec34
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-e5-ef80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:369, threadState:WAITING, lockName:java.lang.Object@186514
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-f5-ff80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:370, threadState:WAITING, lockName:java.lang.Object@19442ab
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-g5-ff80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:371, threadState:WAITING, lockName:java.lang.Object@a51578
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-h5-gf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:372, threadState:WAITING, lockName:java.lang.Object@249d84
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-i5-hf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:373, threadState:WAITING, lockName:java.lang.Object@1e1452e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-j5-hf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:374, threadState:WAITING, lockName:java.lang.Object@85023c
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-k5-if80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:375, threadState:WAITING, lockName:java.lang.Object@55df91
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-l5-if80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:376, threadState:WAITING, lockName:java.lang.Object@1c8d2d2
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-m5-jf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:377, threadState:WAITING, lockName:java.lang.Object@188daee
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-n5-kf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:378, threadState:WAITING, lockName:java.lang.Object@1dd7b8a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-o5-kf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:379, threadState:WAITING, lockName:java.lang.Object@1d9d859
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-p5-lf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:380, threadState:WAITING, lockName:java.lang.Object@10ac851
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-q5-mf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:381, threadState:WAITING, lockName:java.lang.Object@e554f1
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-r5-mf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:382, threadState:WAITING, lockName:java.lang.Object@2baa0f
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-s5-nf80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:383, threadState:WAITING, lockName:java.lang.Object@a9f123
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-t5-of80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:384, threadState:WAITING, lockName:java.lang.Object@1164195
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-u5-of80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:385, threadState:WAITING, lockName:java.lang.Object@2bd941
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-26-sg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:386, threadState:WAITING, lockName:java.lang.Object@c43e97
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-36-tg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:387, threadState:WAITING, lockName:java.lang.Object@1e9686b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-46-vg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:388, threadState:WAITING, lockName:java.lang.Object@c670c3
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-56-xg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:389, threadState:WAITING, lockName:java.lang.Object@151de3
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-66-yg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:390, threadState:WAITING, lockName:java.lang.Object@5d41ec
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-76-yg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:391, threadState:WAITING, lockName:java.lang.Object@931c53
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-86-zg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:392, threadState:WAITING, lockName:java.lang.Object@28c5db
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-96-zg80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:393, threadState:WAITING, lockName:java.lang.Object@d5ec8b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-a6-0h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:394, threadState:WAITING, lockName:java.lang.Object@9cc73d
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-b6-1h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:395, threadState:WAITING, lockName:java.lang.Object@dc98fc
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-c6-1h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:396, threadState:WAITING, lockName:java.lang.Object@bfd10a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-d6-2h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:397, threadState:WAITING, lockName:java.lang.Object@134e8ee
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-e6-3h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:398, threadState:WAITING, lockName:java.lang.Object@a8a280
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-f6-4h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:399, threadState:WAITING, lockName:java.lang.Object@144b834
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-g6-5h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:400, threadState:WAITING, lockName:java.lang.Object@9991dd
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-h6-6h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:401, threadState:WAITING, lockName:java.lang.Object@374580
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          
          Thread: jbm-server-session-i6-7h80uxif-1-z4lutxif-adsehk-t36hs4 : priority:5, demon:false, threadId:402, threadState:WAITING, lockName:java.lang.Object@b3ef3a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:474)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
           java.lang.Thread.run(Thread.java:595)
          


          • 2. Re: JBM thread leak?
            clebert.suconic

            You have a thread (Executor) per Session.

            It looks like you're leaking Sessions.


            As you're using Spring Template (from what I see from the other thread),
            http://www.jboss.com/index.html?module=bb&op=viewtopic&t=139492
            this is highly possible.

            • 3. Re: JBM thread leak?
              theoverlord

              Yes, i'm using the Spring JmsTemplate to send a response in an MDB to a remote clustered queue.

              If I move the MDBs into the same clustered servers where the clustered queues reside, I don't get this problem - it only happens when the MDB is running remotely from the clustered queue.

              Strange.

              From everything I read, it should be safe to use the Spring JmsTemplate for what I'm doing since I'm using it to send a reply and I'm using "java:/JmsXA".

              Doesn't the JmsTemplate close sessions?

              • 4. Re: JBM thread leak?
                clebert.suconic

                 

                "TheOverlord" wrote:
                Yes, i'm using the Spring JmsTemplate to send a response in an MDB to a remote clustered queue.

                If I move the MDBs into the same clustered servers where the clustered queues reside, I don't get this problem - it only happens when the MDB is running remotely from the clustered queue.

                Strange.

                From everything I read, it should be safe to use the Spring JmsTemplate for what I'm doing since I'm using it to send a reply and I'm using "java:/JmsXA".

                Doesn't the JmsTemplate close sessions?


                You need to look at the JmsTemplate source code, or ask that question to Spring guys. We focus ourselves on the JBoss Messaging implementation which is already enough workload for us ;-)

                • 5. Re: JBM thread leak?
                  theoverlord

                  It appears my problems were a Spring bug.

                  I updated my Spring jars from v2.07 to v2.55 and all is well so far with my testing.

                  I think I need a vacation now :)