2 Replies Latest reply on Mar 6, 2009 6:18 PM by clebert.suconic

    Another Deadlock

    clebert.suconic

      Complete thread dump:

       Running org.jboss.messaging.tests.integration.cluster.distribution.NettySymmetricClusterWithDiscoveryTest
      ....
      
      
      Full thread dump Java HotSpot(TM) Server VM (1.5.0_15-b04 mixed mode):
      "Thread-2 (group:JBM-async-session-delivery-threads-25322895)" daemon prio=1 tid=0x6b6aabd8 nid=0xae3 waiting for monitor entry [0x6aa77000..0x6aa780b0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x743a3200> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x743a1910> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x7439edd0> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x7439f110> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-2 (group:JBM-async-session-delivery-threads-17258453)" daemon prio=1 tid=0x63ee2568 nid=0xae2 waiting for monitor entry [0x650b8000..0x650b9030]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x74398000> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x74395c80> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x74390d18> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x74390b48> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-2 (group:JBM-async-session-delivery-threads-727392)" daemon prio=1 tid=0x0a1bf600 nid=0xab9 waiting for monitor entry [0x61e3f000..0x61e3fdb0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x7461b3a0> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x7461b4e0> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x7461b540> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x7461b690> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4336-1" daemon prio=1 tid=0x0a1dbb18 nid=0x92c waiting on condition [0x6264f000..0x62650130]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x7461ccb8> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x7461b3a0> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4337-1" daemon prio=1 tid=0x0a1be680 nid=0x92b waiting on condition [0x627d2000..0x627d30b0]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x7464e868> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x746aa3c8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4335-1" daemon prio=1 tid=0x09e8b900 nid=0x926 waiting on condition [0x683fa000..0x683fae30]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x746bdeb8> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x746c1428> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4333-1" daemon prio=1 tid=0x096f1380 nid=0x923 waiting on condition [0x6b5fe000..0x6b5ff0b0]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x74667f80> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x74668058> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4330-1" daemon prio=1 tid=0x09682ee8 nid=0x922 waiting on condition [0x649aa000..0x649ab030]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x74646608> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x746466e0> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4332-1" daemon prio=1 tid=0x0a3181e8 nid=0x921 waiting on condition [0x4fe99000..0x4fe99fb0]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x746b7218> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x746babf8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4331-1" daemon prio=1 tid=0x09b2d2d8 nid=0x91c waiting on condition [0x5194e000..0x5194edb0]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x746ac180> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x746acfa8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O server worker #195-4" daemon prio=1 tid=0x3eba8fa8 nid=0x917 runnable [0x4fd97000..0x4fd97f30]
       at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
       at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
       at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x74639950> (a sun.nio.ch.Util$1)
       - locked <0x74639940> (a java.util.Collections$UnmodifiableSet)
       - locked <0x7449e3d8> (a sun.nio.ch.PollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-1 (group:JBM-async-session-delivery-threads-25322895)" daemon prio=1 tid=0x09b55840 nid=0x906 waiting for monitor entry [0x4fc95000..0x4fc95eb0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x746c1428> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x746c0a48> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x746bdad8> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x746bdca0> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O server worker #194-3" daemon prio=1 tid=0x3eb56c80 nid=0x900 runnable [0x658c8000..0x658c9030]
       at sun.nio.ch.IOUtil.drain(Native Method)
       at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:66)
       - locked <0x744ac508> (a java.lang.Object)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x744ac4a0> (a sun.nio.ch.Util$1)
       - locked <0x744ac490> (a java.util.Collections$UnmodifiableSet)
       - locked <0x744ac358> (a sun.nio.ch.PollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4320-1" daemon prio=1 tid=0x0a149ad8 nid=0x8fb waiting on condition [0x5001c000..0x5001cdb0]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x74390708> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x74398000> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-1 (group:JBM-async-session-delivery-threads-17258453)" daemon prio=1 tid=0x0a1ce360 nid=0x8f2 waiting for monitor entry [0x51bd3000..0x51bd4130]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x746aa3c8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x746c49b0> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x7464e488> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x7464e650> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "New I/O client worker #4318-1" daemon prio=1 tid=0x09dd5418 nid=0x8f0 waiting on condition [0x50d36000..0x50d37030]
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
       at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
       at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
       - locked <0x7438f9a0> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
       - locked <0x743a3200> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
       at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
       at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
       at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-0 (group:JBM-async-session-delivery-threads-17258453)" daemon prio=1 tid=0x09788d98 nid=0x8eb waiting for monitor entry [0x66561000..0x66561db0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x746babf8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x746ba218> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x746b6e38> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x746b7000> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-0 (group:JBM-async-session-delivery-threads-25322895)" daemon prio=1 tid=0x09df3be0 nid=0x8e9 waiting for monitor entry [0x64bae000..0x64baf0b0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x746466e0> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x74646ff8> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x746d9c00> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x746cb958> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-0 (group:JBM-async-session-delivery-threads-727392)" daemon prio=1 tid=0x0978f388 nid=0x8e7 waiting for monitor entry [0x687fe000..0x687fefb0]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x74668058> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x74668970> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x746d34a0> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x7466abd8> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-1 (group:JBM-async-session-delivery-threads-22216253)" daemon prio=1 tid=0x09edf0e0 nid=0x8e4 waiting for monitor entry [0x3f204000..0x3f204e30]
       at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
       - waiting to lock <0x746acfa8> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
       at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
       at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
       - locked <0x746ac5c8> (a java.lang.Object)
       at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
       at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
       at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
       at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
       at org.jboss.netty.channel.Channels.write(Channels.java:876)
       at org.jboss.netty.channel.Channels.write(Channels.java:824)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
       at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
       - locked <0x746a2840> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
       - locked <0x746a2278> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.ja


        • 1. Re: Another Deadlock
          clebert.suconic

          This Thread is holding a Channel.lock:

          [junit] "Thread-2 (group:JBM-async-session-delivery-threads-17258453)" daemon prio=1 tid=0x63ee2568 nid=0xae2 waiting for monitor entry [0x650b8000..0x650b9030]
           [junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.exceptionCaught(MessagingChannelHandler.java:100)
           [junit] - waiting to lock <0x74398000> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
           [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:224)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:147)
           [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
           [junit] at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:605)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:558)
           [junit] - locked <0x74395c80> (a java.lang.Object)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:322)
           [junit] at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:121)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:792)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:797)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:309)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:271)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:600)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:595)
           [junit] at org.jboss.netty.channel.Channels.write(Channels.java:876)
           [junit] at org.jboss.netty.channel.Channels.write(Channels.java:824)
           [junit] at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
           [junit] at org.jboss.messaging.integration.transports.netty.NettyConnection.write(NettyConnection.java:128)
           [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076)
           [junit] - locked <0x74390d18> (a java.lang.Object)
           [junit] at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:620)
           [junit] at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:263)
           [junit] at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1339)
           [junit] at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:750)
           [junit] at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:322)
           [junit] - locked <0x74390b48> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
           [junit] at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
           [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
           [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
           [junit] at java.lang.Thread.run(Thread.java:595)
          



          If you look at the code, you will see the lock being held:

          org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.sendBlocking(RemotingConnectionImpl.java:1076) (the lock was acquired at lock.lock(), line 1053).


          And this thread is trying to get the channel lock, while it holds 0x74398000

           [junit] "New I/O client worker #4320-1" daemon prio=1 tid=0x0a149ad8 nid=0x8fb waiting on condition [0x5001c000..0x5001cdb0]
           [junit] at sun.misc.Unsafe.park(Native Method)
           [junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
           [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
           [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
           [junit] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:184)
           [junit] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:256)
           [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:547)
           [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
           [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:876)
           [junit] - locked <0x74390708> (a java.lang.Object)
           [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
           [junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.channelDisconnected(MessagingChannelHandler.java:83)
           [junit] - locked <0x74398000> (a org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
           [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:299)
           [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:212)
           [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
           [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
           [junit] at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:494)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:514)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:317)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:306)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
           [junit] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
           [junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
           [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
           [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
           [junit] at java.lang.Thread.run(Thread.java:595)
          
          
          


          • 2. Re: Another Deadlock
            clebert.suconic

             

             [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(Remoting
            ConnectionImpl.java:547)
            
            


            This was introduced on this commit, in an attempt to fix another deadlock.


            http://fisheye.jboss.org/browse/Messaging/trunk/src/main/org/jboss/messaging/core/remoting/impl/RemotingConnectionImpl.java?r1=5967&r2=6014