5 Replies Latest reply on Mar 12, 2009 11:11 PM by clebert.suconic

    Programmatic ThreadDumps on the TestSuite

    clebert.suconic
    clebert.suconic Mar 12, 2009 10:50 AM

      I have added a few calls to a method that I wrote on UnitTestCase (threadDump) around the testsuite.

      In particular, I added one at ClusterTestBase::waitForBindings, and it fired a dump while running the testsuite on my box:


      Most of the threads were idle waiting tasks on the thread pool, but few threads were working around ManagementService:


      -.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526) - this line is waiting for a lock:

      [junit] ===============================================================================
 [junit] Thread Thread[New I/O server worker #198-5,5,jbm-netty-acceptor-worker-threads-292661220] name = New I/O server worker #198-5 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-292661220,maxpri=10]
 [junit]
 [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526)
 [junit] org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.addBinding(PostOfficeImpl.java:510)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleCreateQueue(ServerSessionImpl.java:1340)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCreateQueue(ServerSessionImpl.java:378)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:126)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1486)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:846)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:493)
 [junit] org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
 [junit] org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:73)
 [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
 [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)


      Same thing here: 

      [junit] ===============================================================================
 [junit] Thread Thread[New I/O server worker #198-3,5,jbm-netty-acceptor-worker-threads-292661220] name = New I/O server worker #198-3 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-292661220,maxpri=10]
 [junit]
 [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleCreateConsumer(ServerSessionImpl.java:1262)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCreateConsumer(ServerSessionImpl.java:360)
 [junit] org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:120)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1486)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:846)
 [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:493)
 [junit] org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
 [junit] org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:73)
 [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
 [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================



      And this one is working: 

       [junit] ===============================================================================
 [junit] Thread Thread[Thread-5 (group:jbm-client-session-threads-441502220),5,jbm-client-session-threads-441502220] name = Thread-5 (group:jbm-client-session-threads-441502220) group = java.lang.ThreadGroup[name=jbm-client-session-threads-441502220,maxpri=10]
 [junit]
 [junit] java.lang.Throwable.fillInStackTrace(Native Method)
 [junit] java.lang.Throwable.<init>(Throwable.java:196)
 [junit] java.lang.Exception.<init>(Exception.java:41)
 [junit] java.lang.RuntimeException.<init>(RuntimeException.java:43)
 [junit] java.lang.IndexOutOfBoundsException.<init>(IndexOutOfBoundsException.java:37)
 [junit] java.util.concurrent.CopyOnWriteArrayList.rangeCheck(CopyOnWriteArrayList.java:708)
 [junit] java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:328)
 [junit] org.jboss.messaging.core.postoffice.impl.BindingsImpl.route(BindingsImpl.java:307)
 [junit] org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:635)
 [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:568)
 [junit] org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:784)
 [junit] org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:565)
 [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:564)
 [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl.access$300(ClientConsumerImpl.java:45)
 [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:748)
 [junit] org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================



      I also see several timeouts while this is happening.

      I will make another post with the complete dump. (which is somewhat big (5000 lines))




      • 368 Views
      • Tags:
        • 1. Re: Programmatic ThreadDumps on the TestSuite
          clebert.suconic
          clebert.suconic Mar 12, 2009 10:52 AM (in response to clebert.suconic)

          Here is the complete dump

          (notice this is a programmatic version, so I don't have any information about lock and synchronizations)

          
 [junit] Complete Thread dump
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-598993072),5,jbm-netty-connector-boss-threads-598993072] name = Thread-0 (group:jbm-netty-connector-boss-threads-598993072) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-598993072,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-691894544),5,jbm-netty-connector-boss-threads-691894544] name = Thread-0 (group:jbm-netty-connector-boss-threads-691894544) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-691894544,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-12444689),5,jbm-netty-connector-boss-threads-12444689] name = Thread-0 (group:jbm-netty-connector-boss-threads-12444689) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-12444689,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1665641200),5,jbm-netty-connector-boss-threads-1665641200] name = Thread-0 (group:jbm-netty-connector-boss-threads-1665641200) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1665641200,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-701454880),5,jbm-netty-connector-boss-threads-701454880] name = Thread-0 (group:jbm-netty-connector-boss-threads-701454880) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-701454880,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-157646098),5,jbm-netty-connector-boss-threads-157646098] name = Thread-0 (group:jbm-netty-connector-boss-threads-157646098) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-157646098,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-113062904),5,jbm-netty-connector-boss-threads-113062904] name = Thread-0 (group:jbm-netty-connector-boss-threads-113062904) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-113062904,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1214828746),5,jbm-netty-connector-boss-threads-1214828746] name = Thread-0 (group:jbm-netty-connector-boss-threads-1214828746) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1214828746,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-222086435),5,jbm-netty-connector-boss-threads-222086435] name = Thread-0 (group:jbm-netty-connector-boss-threads-222086435) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-222086435,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:JBM-async-session-delivery-threads-644010817),5,JBM-async-session-delivery-threads-644010817] name = Thread-0 (group:JBM-async-session-delivery-threads-644010817) group = java.lang.ThreadGroup[name=JBM-async-session-delivery-threads-644010817,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[New I/O server worker #197-1,5,jbm-netty-acceptor-worker-threads-1200332508] name = New I/O server worker #197-1 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-1200332508,maxpri=10]
 [junit]
 [junit] sun.nio.ch.PollArrayWrapper.poll0(Native Method)
 [junit] sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
 [junit] sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
 [junit] sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
 [junit] sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1737011065),5,jbm-netty-connector-boss-threads-1737011065] name = Thread-0 (group:jbm-netty-connector-boss-threads-1737011065) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1737011065,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1507980599),5,jbm-netty-connector-boss-threads-1507980599] name = Thread-0 (group:jbm-netty-connector-boss-threads-1507980599) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1507980599,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-192678515),5,jbm-netty-connector-boss-threads-192678515] name = Thread-0 (group:jbm-netty-connector-boss-threads-192678515) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-192678515,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-4845,5,main] name = Thread-4845 group = java.lang.ThreadGroup[name=main,maxpri=10]
 [junit]
 [junit] java.net.PlainDatagramSocketImpl.receive0(Native Method)
 [junit] java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
 [junit] java.net.DatagramSocket.receive(DatagramSocket.java:712)
 [junit] org.jboss.messaging.core.cluster.impl.DiscoveryGroupImpl.run(DiscoveryGroupImpl.java:213)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1247725687),5,jbm-netty-connector-boss-threads-1247725687] name = Thread-0 (group:jbm-netty-connector-boss-threads-1247725687) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1247725687,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-4854,5,main] name = Thread-4854 group = java.lang.ThreadGroup[name=main,maxpri=10]
 [junit]
 [junit] java.net.PlainDatagramSocketImpl.receive0(Native Method)
 [junit] java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
 [junit] java.net.DatagramSocket.receive(DatagramSocket.java:712)
 [junit] org.jboss.messaging.core.cluster.impl.DiscoveryGroupImpl.run(DiscoveryGroupImpl.java:213)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-580554322),5,jbm-netty-connector-boss-threads-580554322] name = Thread-0 (group:jbm-netty-connector-boss-threads-580554322) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-580554322,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[New I/O server boss #198 (channelId: 544680888, /127.0.0.1:5447),5,jbm-netty-acceptor-boss-threads-2061893037] name = New I/O server boss #198 (channelId: 544680888, /127.0.0.1:5447) group = java.lang.ThreadGroup[name=jbm-netty-acceptor-boss-threads-2061893037,maxpri=10]
 [junit]
 [junit] sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
 [junit] sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
 [junit] org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:204)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1311743684),5,jbm-netty-connector-boss-threads-1311743684] name = Thread-0 (group:jbm-netty-connector-boss-threads-1311743684) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1311743684,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-837452093),5,jbm-netty-connector-boss-threads-837452093] name = Thread-0 (group:jbm-netty-connector-boss-threads-837452093) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-837452093,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-worker-threads-369281121),5,jbm-netty-connector-worker-threads-369281121] name = Thread-0 (group:jbm-netty-connector-worker-threads-369281121) group = java.lang.ThreadGroup[name=jbm-netty-connector-worker-threads-369281121,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1563883839),5,jbm-netty-connector-boss-threads-1563883839] name = Thread-0 (group:jbm-netty-connector-boss-threads-1563883839) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1563883839,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-2144736082),5,jbm-netty-connector-boss-threads-2144736082] name = Thread-0 (group:jbm-netty-connector-boss-threads-2144736082) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-2144736082,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1008607187),5,jbm-netty-connector-boss-threads-1008607187] name = Thread-0 (group:jbm-netty-connector-boss-threads-1008607187) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1008607187,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-acceptor-worker-threads-188420353),5,jbm-netty-acceptor-worker-threads-188420353] name = Thread-0 (group:jbm-netty-acceptor-worker-threads-188420353) group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-188420353,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-340360870),5,jbm-netty-connector-boss-threads-340360870] name = Thread-0 (group:jbm-netty-connector-boss-threads-340360870) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-340360870,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-acceptor-worker-threads-315780191),5,jbm-netty-acceptor-worker-threads-315780191] name = Thread-0 (group:jbm-netty-acceptor-worker-threads-315780191) group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-315780191,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:JBM-scheduled-threads-2085136083),5,JBM-scheduled-threads-2085136083] name = Thread-0 (group:JBM-scheduled-threads-2085136083) group = java.lang.ThreadGroup[name=JBM-scheduled-threads-2085136083,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
 [junit] java.util.concurrent.DelayQueue.take(DelayQueue.java:135)
 [junit] java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
 [junit] java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-553434543),5,jbm-netty-connector-boss-threads-553434543] name = Thread-0 (group:jbm-netty-connector-boss-threads-553434543) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-553434543,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1750362802),5,jbm-netty-connector-boss-threads-1750362802] name = Thread-0 (group:jbm-netty-connector-boss-threads-1750362802) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1750362802,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-170347804),5,jbm-netty-connector-boss-threads-170347804] name = Thread-0 (group:jbm-netty-connector-boss-threads-170347804) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-170347804,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-worker-threads-1191069340),5,jbm-netty-connector-worker-threads-1191069340] name = Thread-0 (group:jbm-netty-connector-worker-threads-1191069340) group = java.lang.ThreadGroup[name=jbm-netty-connector-worker-threads-1191069340,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1309194397),5,jbm-netty-connector-boss-threads-1309194397] name = Thread-0 (group:jbm-netty-connector-boss-threads-1309194397) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1309194397,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-2062869828),5,jbm-netty-connector-boss-threads-2062869828] name = Thread-0 (group:jbm-netty-connector-boss-threads-2062869828) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-2062869828,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[New I/O server worker #196-6,5,jbm-netty-acceptor-worker-threads-2146808717] name = New I/O server worker #196-6 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-2146808717,maxpri=10]
 [junit]
 [junit] sun.nio.ch.PollArrayWrapper.poll0(Native Method)
 [junit] sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
 [junit] sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
 [junit] sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
 [junit] sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:JBM-async-session-delivery-threads-963172827),5,JBM-async-session-delivery-threads-963172827] name = Thread-0 (group:JBM-async-session-delivery-threads-963172827) group = java.lang.ThreadGroup[name=JBM-async-session-delivery-threads-963172827,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[New I/O client worker #2585-1,5,jbm-netty-connector-worker-threads-1171514533] name = New I/O client worker #2585-1 group = java.lang.ThreadGroup[name=jbm-netty-connector-worker-threads-1171514533,maxpri=10]
 [junit]
 [junit] sun.nio.ch.PollArrayWrapper.poll0(Native Method)
 [junit] sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
 [junit] sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
 [junit] sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
 [junit] sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:JBM-scheduled-threads-387688567),5,JBM-scheduled-threads-387688567] name = Thread-0 (group:JBM-scheduled-threads-387688567) group = java.lang.ThreadGroup[name=JBM-scheduled-threads-387688567,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
 [junit] java.util.concurrent.DelayQueue.take(DelayQueue.java:135)
 [junit] java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
 [junit] java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-437836513),5,jbm-netty-connector-boss-threads-437836513] name = Thread-0 (group:jbm-netty-connector-boss-threads-437836513) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-437836513,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-2 (group:JBM-async-session-delivery-threads-1351806718),5,JBM-async-session-delivery-threads-1351806718] name = Thread-2 (group:JBM-async-session-delivery-threads-1351806718) group = java.lang.ThreadGroup[name=JBM-async-session-delivery-threads-1351806718,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1850345548),5,jbm-netty-connector-boss-threads-1850345548] name = Thread-0 (group:jbm-netty-connector-boss-threads-1850345548) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1850345548,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[New I/O client worker #2587-1,5,jbm-netty-connector-worker-threads-1706744318] name = New I/O client worker #2587-1 group = java.lang.ThreadGroup[name=jbm-netty-connector-worker-threads-1706744318,maxpri=10]
 [junit]
 [junit] sun.nio.ch.PollArrayWrapper.poll0(Native Method)
 [junit] sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
 [junit] sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
 [junit] sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
 [junit] sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
 [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:158)
 [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-753923815),5,jbm-netty-connector-boss-threads-753923815] name = Thread-0 (group:jbm-netty-connector-boss-threads-753923815) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-753923815,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-748666335),5,jbm-netty-connector-boss-threads-748666335] name = Thread-0 (group:jbm-netty-connector-boss-threads-748666335) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-748666335,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1679796651),5,jbm-netty-connector-boss-threads-1679796651] name = Thread-0 (group:jbm-netty-connector-boss-threads-1679796651) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1679796651,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1275914301),5,jbm-netty-connector-boss-threads-1275914301] name = Thread-0 (group:jbm-netty-connector-boss-threads-1275914301) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1275914301,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1708403061),5,jbm-netty-connector-boss-threads-1708403061] name = Thread-0 (group:jbm-netty-connector-boss-threads-1708403061) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1708403061,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1482553286),5,jbm-netty-connector-boss-threads-1482553286] name = Thread-0 (group:jbm-netty-connector-boss-threads-1482553286) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1482553286,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1209225958),5,jbm-netty-connector-boss-threads-1209225958] name = Thread-0 (group:jbm-netty-connector-boss-threads-1209225958) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1209225958,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-503671162),5,jbm-netty-connector-boss-threads-503671162] name = Thread-0 (group:jbm-netty-connector-boss-threads-503671162) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-503671162,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1057793548),5,jbm-netty-connector-boss-threads-1057793548] name = Thread-0 (group:jbm-netty-connector-boss-threads-1057793548) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1057793548,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-1 (group:JBM-async-session-delivery-threads-539825119),5,JBM-async-session-delivery-threads-539825119] name = Thread-1 (group:JBM-async-session-delivery-threads-539825119) group = java.lang.ThreadGroup[name=JBM-async-session-delivery-threads-539825119,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-292611174),5,jbm-netty-connector-boss-threads-292611174] name = Thread-0 (group:jbm-netty-connector-boss-threads-292611174) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-292611174,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-1273657593),5,jbm-netty-connector-boss-threads-1273657593] name = Thread-0 (group:jbm-netty-connector-boss-threads-1273657593) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-1273657593,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-742784538),5,jbm-netty-connector-boss-threads-742784538] name = Thread-0 (group:jbm-netty-connector-boss-threads-742784538) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-742784538,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
 [junit] java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
 [junit] java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
 [junit] java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
 [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
 [junit] java.lang.Thread.run(Thread.java:595)
 [junit] ===============================================================================
 [junit] Thread Thread[Thread-0 (group:jbm-netty-connector-boss-threads-301036308),5,jbm-netty-connector-boss-threads-301036308] name = Thread-0 (group:jbm-netty-connector-boss-threads-301036308) group = java.lang.ThreadGroup[name=jbm-netty-connector-boss-threads-301036308,maxpri=10]
 [junit]
 [junit] sun.misc.Unsafe.park(Native Method)
 [junit] java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
 [junit] java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchro


            • Actions
          • 2. Re: Programmatic ThreadDumps on the TestSuite
            clebert.suconic
            clebert.suconic Mar 12, 2009 12:50 PM (in response to clebert.suconic)

            The root cause is an infinite loop BindingsImpl::route, that will probably start when the binding is deleted in the middle of the routing.

            I added some debug code into BindingsImpl::route :

            long start = System.currentTimeMillis();

 long lastPrint = System.currentTimeMillis();

 int nexceptions = 0;
 while (true)
 {
 Binding binding;
 try
 {
 if (System.currentTimeMillis() - lastPrint > 1000)
 {
 lastPrint = System.currentTimeMillis();
 System.out.println("-------------------------------> On BindingsImpl::route loop for " + (System.currentTimeMillis() - start) + " milliseconds now, pos = " +
 "" + pos + " bindings.size = " + bindings.size());
 }
 binding = bindings.get(pos);
 }
 catch (IndexOutOfBoundsException e)
 {
 // This can occur if binding is removed while in route
 if (!bindings.isEmpty())
 {
 pos = 0;

 continue;
 }
 else
 {
 break;
 }
 }


            And the loop stays running forever, even after the test is finished, probably affecting next tests.

            I attached a debugger, and the only Binding available, has a filter which won't match (probably that special filter for empty Addresses?)

            The test stays on a loop, getting back to the IndexOutOfBoundsException.. goes back to 0, find the binding... and then it tries to match the filter again, going back to the Exception... and again, and again, and again...

              • Actions
            • 3. Re: Programmatic ThreadDumps on the TestSuite
              clebert.suconic
              clebert.suconic Mar 12, 2009 1:29 PM (in response to clebert.suconic)

              https://jira.jboss.org/jira/browse/JBMESSAGING-1545

                • Actions
              • 4. Re: Programmatic ThreadDumps on the TestSuite
                clebert.suconic
                clebert.suconic Mar 12, 2009 3:07 PM (in response to clebert.suconic)

                I have committed a hack, so just the infinite loop is gone.. just to see how Hudson would behave. I did a run at my box and I got a successful run.

                Tim: if you could please take a look, and find a better fix.

                  • Actions
                • 5. Re: Programmatic ThreadDumps on the TestSuite
                  clebert.suconic
                  clebert.suconic Mar 12, 2009 11:11 PM (in response to clebert.suconic)

                  I have removed the Hack and proposed a better fix now. I' m just reseting the counters and sizes properly. (Tim: Please take a look)

                  I also have replicated this on a new unit test:
                  org.jboss.messaging.tests.unit.core.postoffice.impl.BindingImplTest


                  If you revert BindingsImpl to 5967 the test will loop the same way that happened earlier on the testsuite.

                  I' m testing the same scenario on both route and redistribute.

                  BTW: It' s quite good seeing the testsuite behaving more reliably. We still have a few failures in Bridge... but we are getting there.

                    • Actions