5 Replies Latest reply on Aug 18, 2009 7:18 AM by Tim Fox

    Bridge Leaking

    Clebert Suconic Master

      As I suspected, BridgeImpl is leaking. If we can fix this we would speed up the testsuite.

      This is easily replicated with MessageRedistributionTest::testRedistributionWhenConsumerIsClosed.

      (Probably with the others Redistribution tests also, but this one above is the test I was using to debug it).


      I - At the beggining, MessageRedistributionTest will setup three servers, with bridges between all of them.

      II - When the test is finished, tearDown will stop Server 0, 1 and 2.

      III - Server0 is stopped successfully without any problems.

      IV - When server1 is stopped, the bridge between server1 and server0 will try a session.close(). Session.close will close the consumers which will never succeed, being hanging at this point on StopRunnable:

      Thread-4 (group:JBM-server-threads1809605797-1381221365)" prio=10 tid=0x00007f80a81af400 nid=0x3a0d waiting on condition [0x00007f809fffe000..0x00007f809fffec70]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80e0600878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
       at org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:292)
       - locked <0x00007f80e06008d8> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:745)
       at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:315)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.closeChildren(ClientSessionImpl.java:1322)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:708)
       at org.jboss.messaging.core.client.impl.DelegatingSession.close(DelegatingSession.java:116)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:702)
       - locked <0x00007f80e057df00> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      


      This thread will hung here even after the server is finished.

      * I will provide the complete thread dump at the end of this post.


      V - The test will finish allright, however ClientSession from the Bridge will still reference the Server, and threads/executors will just be hunging around.


      VI - As you can see also, there are several threads trying to reconnect to the server: look at the complete dump:

      Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.3-b02 mixed mode):
      
      "Thread-30" prio=10 tid=0x00007f80a8088800 nid=0x3afb runnable [0x00007f80ae04c000..0x00007f80ae04c9f0]
       java.lang.Thread.State: RUNNABLE
       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x00007f80de8fdf98> (a sun.nio.ch.Util$1)
       - locked <0x00007f80de8fdf80> (a java.util.Collections$UnmodifiableSet)
       - locked <0x00007f80de8fdc20> (a sun.nio.ch.EPollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:289)
       at java.lang.Thread.run(Thread.java:619)
      
      "server-timer" daemon prio=10 tid=0x00007f80a8086c00 nid=0x3afa in Object.wait() [0x00007f80ada46000..0x00007f80ada46a70]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00007f80de8fe5f0> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x00007f80de8fe5f0> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "Thread-5 (group:JBM-server-threads1809605797-1381221365)" prio=10 tid=0x00007f80a81b0c00 nid=0x3aed waiting for monitor entry [0x00007f80ac935000..0x00007f80ac935b70]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:435)
       - waiting to lock <0x00007f80b5d8fe88> (a java.lang.Object)
       - locked <0x00007f80b5d8fe98> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1296)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:719)
       at org.jboss.messaging.core.client.impl.DelegatingSession.close(DelegatingSession.java:116)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:702)
       - locked <0x00007f80b5d90030> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-4 (group:JBM-server-threads1809605797-1381221365)" prio=10 tid=0x00007f80a81af400 nid=0x3a0d waiting for monitor entry [0x00007f809fffe000..0x00007f809fffec70]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:435)
       - waiting to lock <0x00007f80b5d90708> (a java.lang.Object)
       - locked <0x00007f80b5d90718> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1296)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:719)
       at org.jboss.messaging.core.client.impl.DelegatingSession.close(DelegatingSession.java:116)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:702)
       - locked <0x00007f80b5d908b0> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-5 (group:JBM-client-global-threads-799449511)" daemon prio=10 tid=0x0000000042166c00 nid=0x39e1 waiting on condition [0x00007f80adb47000..0x00007f80adb47cf0]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
       at java.lang.Thread.sleep(Native Method)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.getConnectionWithRetry(ConnectionManagerImpl.java:790)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:710)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:611)
       - locked <0x00007f80b5d90708> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:483)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$2(ConnectionManagerImpl.java:481)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1151)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:434)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$Channel0Handler$1.run(ConnectionManagerImpl.java:1094)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-4 (group:JBM-server-threads2023202663-711090443)" prio=10 tid=0x0000000042182400 nid=0x39a2 waiting for monitor entry [0x00007f80ac32f000..0x00007f80ac32fbf0]
       java.lang.Thread.State: BLOCKED (on object monitor)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:435)
       - waiting to lock <0x00007f80b5e1b878> (a java.lang.Object)
       - locked <0x00007f80b5e1b888> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1296)
       at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:719)
       at org.jboss.messaging.core.client.impl.DelegatingSession.close(DelegatingSession.java:116)
       at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$StopRunnable.run(BridgeImpl.java:702)
       - locked <0x00007f80b5df7598> (a org.jboss.messaging.core.server.cluster.impl.BridgeImpl)
       at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-4 (group:JBM-client-global-scheduled-threads-1690235745)" daemon prio=10 tid=0x00007f80a81a6c00 nid=0x39a1 waiting on condition [0x00007f80ac430000..0x00007f80ac430c70]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80b5d98828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-3 (group:JBM-client-global-threads-799449511)" daemon prio=10 tid=0x000000004216e400 nid=0x399e waiting on condition [0x00007f80ac531000..0x00007f80ac5319f0]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
       at java.lang.Thread.sleep(Native Method)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.getConnectionWithRetry(ConnectionManagerImpl.java:790)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:710)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:611)
       - locked <0x00007f80b5d8fe88> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:483)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$2(ConnectionManagerImpl.java:481)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1151)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:434)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$Channel0Handler$1.run(ConnectionManagerImpl.java:1094)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-0 (group:JBM-client-global-threads-799449511)" daemon prio=10 tid=0x0000000042137400 nid=0x3997 waiting on condition [0x00007f80acc38000..0x00007f80acc38d70]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
       at java.lang.Thread.sleep(Native Method)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.getConnectionWithRetry(ConnectionManagerImpl.java:790)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.reattachSessions(ConnectionManagerImpl.java:710)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:611)
       - locked <0x00007f80b5e1b878> (a java.lang.Object)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:483)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$2(ConnectionManagerImpl.java:481)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1151)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:434)
       at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)
       at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$Channel0Handler$1.run(ConnectionManagerImpl.java:1094)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-3 (group:JBM-client-global-scheduled-threads-1690235745)" daemon prio=10 tid=0x00007f80a819c400 nid=0x3992 waiting on condition [0x00007f80ad13d000..0x00007f80ad13dbf0]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80b5d98828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-2 (group:JBM-client-global-scheduled-threads-1690235745)" daemon prio=10 tid=0x0000000042141c00 nid=0x3991 waiting on condition [0x00007f80ad23e000..0x00007f80ad23ec70]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80b5d98828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-1 (group:JBM-client-global-scheduled-threads-1690235745)" daemon prio=10 tid=0x00007f80a406fc00 nid=0x3990 waiting on condition [0x00007f80ad33f000..0x00007f80ad33fcf0]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80b5d98828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-0 (group:JBM-client-global-scheduled-threads-1690235745)" daemon prio=10 tid=0x00007f80a406e800 nid=0x398f waiting on condition [0x00007f80ad440000..0x00007f80ad440d70]
       java.lang.Thread.State: TIMED_WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80b5d98828> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
       at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
       at java.lang.Thread.run(Thread.java:619)
      
      "Thread-2" prio=10 tid=0x00007f80a026e400 nid=0x397b runnable [0x00007f80ae85a000..0x00007f80ae85aaf0]
       java.lang.Thread.State: RUNNABLE
       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x00007f80b5d9bff0> (a sun.nio.ch.Util$1)
       - locked <0x00007f80b5d9bfd8> (a java.util.Collections$UnmodifiableSet)
       - locked <0x00007f80b5d9bc00> (a sun.nio.ch.EPollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:289)
       at java.lang.Thread.run(Thread.java:619)
      
      "server-timer" daemon prio=10 tid=0x00007f80a0263000 nid=0x3979 in Object.wait() [0x00007f80ae95b000..0x00007f80ae95bb70]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00007f80b5d9c7c0> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x00007f80b5d9c7c0> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "ReaderThread" prio=10 tid=0x00007f80a0270000 nid=0x3977 runnable [0x00007f80aeb64000..0x00007f80aeb64bf0]
       java.lang.Thread.State: RUNNABLE
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
       at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
       at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
       - locked <0x00007f80b5ca74d0> (a java.io.InputStreamReader)
       at java.io.InputStreamReader.read(InputStreamReader.java:167)
       at java.io.BufferedReader.fill(BufferedReader.java:136)
       at java.io.BufferedReader.readLine(BufferedReader.java:299)
       - locked <0x00007f80b5ca74d0> (a java.io.InputStreamReader)
       at java.io.BufferedReader.readLine(BufferedReader.java:362)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner$ReaderThread.run(RemoteTestRunner.java:140)
      
      "Low Memory Detector" daemon prio=10 tid=0x00007f80a0002000 nid=0x3974 runnable [0x0000000000000000..0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
      
      "CompilerThread1" daemon prio=10 tid=0x00007f80a8046400 nid=0x3973 waiting on condition [0x0000000000000000..0x00007f80af145570]
       java.lang.Thread.State: RUNNABLE
      
      "CompilerThread0" daemon prio=10 tid=0x00007f80a8044800 nid=0x3972 waiting on condition [0x0000000000000000..0x00007f80af2461f0]
       java.lang.Thread.State: RUNNABLE
      
      "JDWP Command Reader" daemon prio=10 tid=0x000000004209ac00 nid=0x3970 runnable [0x0000000000000000..0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
      
      "JDWP Event Helper Thread" daemon prio=10 tid=0x00007f80a802f400 nid=0x396e runnable [0x0000000000000000..0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
      
      "JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x00007f80a802bc00 nid=0x396c runnable [0x0000000000000000..0x00007f80af54a750]
       java.lang.Thread.State: RUNNABLE
      
      "Signal Dispatcher" daemon prio=10 tid=0x00007f80a801cc00 nid=0x396a waiting on condition [0x0000000000000000..0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
      
      "Finalizer" daemon prio=10 tid=0x0000000042093800 nid=0x3969 in Object.wait() [0x00007f80af850000..0x00007f80af850c70]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00007f80b5cc28f8> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
       - locked <0x00007f80b5cc28f8> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0x0000000042091800 nid=0x3968 in Object.wait() [0x00007f80af951000..0x00007f80af951cf0]
       java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       - waiting on <0x00007f80b5cc2930> (a java.lang.ref.Reference$Lock)
       at java.lang.Object.wait(Object.java:485)
       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
       - locked <0x00007f80b5cc2930> (a java.lang.ref.Reference$Lock)
      
      "main" prio=10 tid=0x0000000042020c00 nid=0x3959 waiting on condition [0x00007f80f7732000..0x00007f80f7732eb0]
       java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for <0x00007f80de8fd208> (a java.util.concurrent.CountDownLatch$Sync)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
       at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
       at org.jboss.profiler.jvmti.JVMTIInterface.startMemoryProfiler(JVMTIInterface.java:975)
       at org.jboss.messaging.tests.util.UnitTestCase.tearDown(UnitTestCase.java:617)
       at org.jboss.messaging.tests.integration.cluster.distribution.ClusterTestBase.tearDown(ClusterTestBase.java:114)
       at org.jboss.messaging.tests.integration.cluster.distribution.MessageRedistributionTest.tearDown(MessageRedistributionTest.java:67)
       at junit.framework.TestCase.runBare(TestCase.java:136)
       at junit.framework.TestResult$1.protect(TestResult.java:106)
       at junit.framework.TestResult.runProtected(TestResult.java:124)
       at junit.framework.TestResult.run(TestResult.java:109)
       at junit.framework.TestCase.run(TestCase.java:120)
       at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
       at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
       at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      
      "VM Thread" prio=10 tid=0x000000004208c000 nid=0x3967 runnable
      
      "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004202b800 nid=0x395f runnable
      
      "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004202d400 nid=0x3960 runnable
      
      "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000004202ec00 nid=0x3961 runnable
      
      "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000042030800 nid=0x3962 runnable
      
      "GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000042032000 nid=0x3963 runnable
      
      "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000042033800 nid=0x3964 runnable
      
      "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000042035400 nid=0x3965 runnable
      
      "GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000042036c00 nid=0x3966 runnable
      
      "VM Periodic Task Thread" prio=10 tid=0x00007f80a0004800 nid=0x3975 waiting on condition
      
      JNI global references: 31824
      
      Heap
       PSYoungGen total 31424K, used 2359K [0x00007f80de720000, 0x00007f80e0ac0000, 0x00007f80f2c70000)
       eden space 28864K, 8% used [0x00007f80de720000,0x00007f80de96de48,0x00007f80e0350000)
       from space 2560K, 0% used [0x00007f80e0350000,0x00007f80e0350000,0x00007f80e05d0000)
       to space 2816K, 0% used [0x00007f80e0800000,0x00007f80e0800000,0x00007f80e0ac0000)
       PSOldGen total 77056K, used 2030K [0x00007f80b5c70000, 0x00007f80ba7b0000, 0x00007f80de720000)
       object space 77056K, 2% used [0x00007f80b5c70000,0x00007f80b5e6b8a8,0x00007f80ba7b0000)
       PSPermGen total 21248K, used 10560K [0x00007f80b0870000, 0x00007f80b1d30000, 0x00007f80b5c70000)
       object space 21248K, 49% used [0x00007f80b0870000,0x00007f80b12c0288,0x00007f80b1d30000)