2 Replies Latest reply on Nov 21, 2007 9:34 AM by Travis Brown

    Deadlocks with ClusterRoundRobinDistributor

    Travis Brown Newbie

      During volume testing with 2 nodes I now see these deadlocks in the thread dumps (platform AS 4.2.0.GA_CP01, JBM snapshot_stable 12-Nov-2007):

      Node 1:

      Found one Java-level deadlock:
      =============================
      "Messaging Timeout(3)-266":
       waiting to lock monitor 0x7548e274 (object 0x83f80fa0, a java.lang.Object),
       which is held by "Thread-114"
      "Thread-114":
       waiting to lock monitor 0x7a7dc6e4 (object 0x83f80f70, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
       which is held by "Messaging Timeout(3)-266"
      
      Java stack information for the threads listed above:
      ===================================================
      "Messaging Timeout(3)-266":
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
       - waiting to lock <0x83f80fa0> (a java.lang.Object)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - locked <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
       at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
       at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      "Thread-114":
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - waiting to lock <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
       - locked <0x83f80fa0> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
       at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
       at java.lang.Thread.run(Thread.java:595)
      
      Found one Java-level deadlock:
      =============================
      "Thread-424":
       waiting to lock monitor 0x7548e2f4 (object 0x8409cda0, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
       which is held by "Messaging Timeout(3)-281"
      "Messaging Timeout(3)-281":
       waiting to lock monitor 0x7548e874 (object 0x8409cdd0, a java.lang.Object),
       which is held by "Thread-424"
      
      Java stack information for the threads listed above:
      ===================================================
      "Thread-424":
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - waiting to lock <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
       - locked <0x8409cdd0> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
       at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
       at java.lang.Thread.run(Thread.java:595)
      "Messaging Timeout(3)-281":
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
       - waiting to lock <0x8409cdd0> (a java.lang.Object)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - locked <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
       at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
       at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      
      Found 2 deadlocks.
      


      Node 2:
      Found one Java-level deadlock:
      =============================
      "Thread-107":
       waiting to lock monitor 0x08355a8c (object 0x841ad4d8, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor),
       which is held by "Messaging Timeout(3)-13"
      "Messaging Timeout(3)-13":
       waiting to lock monitor 0x087f6f54 (object 0x841ad508, a java.lang.Object),
       which is held by "Thread-107"
      
      Java stack information for the threads listed above:
      ===================================================
      "Thread-107":
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - waiting to lock <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222)
       - locked <0x841ad508> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
       at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
       at java.lang.Thread.run(Thread.java:595)
      "Messaging Timeout(3)-13":
       at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621)
       - waiting to lock <0x841ad508> (a java.lang.Object)
       at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129)
       - locked <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor)
       at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956)
       at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265)
       at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      
      Found 1 deadlock.
      


      My message processing comes to a halt, but I doubt it's fully related to these 6 threads.