Deadlocks with ClusterRoundRobinDistributor
During volume testing with 2 nodes I now see these deadlocks in the thread dumps (platform AS 4.2.0.GA_CP01, JBM snapshot_stable 12-Nov-2007):
Node 1:
Found one Java-level deadlock: ============================= "Messaging Timeout(3)-266": waiting to lock monitor 0x7548e274 (object 0x83f80fa0, a java.lang.Object), which is held by "Thread-114" "Thread-114": waiting to lock monitor 0x7a7dc6e4 (object 0x83f80f70, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor), which is held by "Messaging Timeout(3)-266" Java stack information for the threads listed above: =================================================== "Messaging Timeout(3)-266": at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621) - waiting to lock <0x83f80fa0> (a java.lang.Object) at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - locked <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956) at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743) at java.lang.Thread.run(Thread.java:595) "Thread-114": at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - waiting to lock <0x83f80f70> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222) - locked <0x83f80fa0> (a java.lang.Object) at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598) at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89) at java.lang.Thread.run(Thread.java:595) Found one Java-level deadlock: ============================= "Thread-424": waiting to lock monitor 0x7548e2f4 (object 0x8409cda0, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor), which is held by "Messaging Timeout(3)-281" "Messaging Timeout(3)-281": waiting to lock monitor 0x7548e874 (object 0x8409cdd0, a java.lang.Object), which is held by "Thread-424" Java stack information for the threads listed above: =================================================== "Thread-424": at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - waiting to lock <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222) - locked <0x8409cdd0> (a java.lang.Object) at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598) at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89) at java.lang.Thread.run(Thread.java:595) "Messaging Timeout(3)-281": at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621) - waiting to lock <0x8409cdd0> (a java.lang.Object) at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - locked <0x8409cda0> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956) at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743) at java.lang.Thread.run(Thread.java:595) Found 2 deadlocks.
Node 2:
Found one Java-level deadlock: ============================= "Thread-107": waiting to lock monitor 0x08355a8c (object 0x841ad4d8, a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor), which is held by "Messaging Timeout(3)-13" "Messaging Timeout(3)-13": waiting to lock monitor 0x087f6f54 (object 0x841ad508, a java.lang.Object), which is held by "Thread-107" Java stack information for the threads listed above: =================================================== "Thread-107": at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - waiting to lock <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:222) - locked <0x841ad508> (a java.lang.Object) at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598) at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89) at java.lang.Thread.run(Thread.java:595) "Messaging Timeout(3)-13": at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.getNumberOfReceivers(MessagingQueue.java:621) - waiting to lock <0x841ad508> (a java.lang.Object) at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.getNumberOfReceivers(ClusterRoundRobinDistributor.java:129) - locked <0x841ad4d8> (a org.jboss.messaging.core.impl.ClusterRoundRobinDistributor) at org.jboss.messaging.core.impl.ChannelSupport$DeliverRefTimeoutTarget.timedOut(ChannelSupport.java:956) at org.jboss.util.timeout.TimeoutFactory$TimeoutWorker.run(TimeoutFactory.java:265) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743) at java.lang.Thread.run(Thread.java:595) Found 1 deadlock.
My message processing comes to a halt, but I doubt it's fully related to these 6 threads.
