1 2 Previous Next 17 Replies Latest reply on Mar 5, 2009 9:55 AM by ataylor

    Another DeadLock, something around InVM

    clebert.suconic
    clebert.suconic Mar 4, 2009 1:20 PM

      *maybe* related to JORAM hangs:

       [junit] Found one Java-level deadlock:
 [junit] =============================
 [junit] "Thread-11705":
 [junit] waiting to lock monitor 0xa8838124 (object 0xb276fcd8, a java.lang.Object),
 [junit] which is held by "Thread-11703"
 [junit] "Thread-11703":
 [junit] waiting to lock monitor 0xa88381e4 (object 0xb279d280, a org.jboss.messaging.core.remoting.impl.invm.InVMConnection),
 [junit] which is held by "Thread-11705"
 [junit]
 [junit] Java stack information for the threads listed above:
 [junit] ===================================================
 [junit] "Thread-11705":
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:867)
 [junit] - waiting to lock <0xb276fcd8> (a java.lang.Object)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector$Listener.connectionDestroyed(InVMConnector.java:187)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:96)
 [junit] - locked <0xb279d280> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector.disconnect(InVMConnector.java:161)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor$Listener$1.run(InVMAcceptor.java:167)
 [junit] "Thread-11703":
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:86)
 [junit] - waiting to lock <0xb279d280> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
 [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:557)
 [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:423)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:871)
 [junit] - locked <0xb276fcd8> (a java.lang.Object)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector$Listener.connectionDestroyed(InVMConnector.java:187)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:96)
 [junit] - locked <0xb2775b98> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector.disconnect(InVMConnector.java:161)
 [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor$Listener$1.run(InVMAcceptor.java:167)


      • 891 Views
      • Tags:
        • 1. Re: Another DeadLock, something around InVM
          clebert.suconic
          clebert.suconic Mar 4, 2009 3:06 PM (in response to clebert.suconic)

          I was trying to replicate this on a test.

          I came up with a test, but it hangs with a different thread dump.


          it would probably be a good test.

          It starts a bunch of threads, all of them missing pings, and all of them closing the session.

          I'm playing with a random wait between the createSession and close:

          Thread.sleep(PING_INTERVAL * (RandomUtil.randomPositiveInt() % 3));


          So, I could eventually have session.close and failure detection being called at the same time.


          WDYT ? (is that a valid test?):




          public void testMultiThreadOpenAndCloses() throws Exception
{
 final TransportConfiguration transportConfig = new TransportConfiguration("org.jboss.messaging.integration.transports.netty.NettyConnectorFactory");

 Interceptor noPongInterceptor = new Interceptor()
 {
 public boolean intercept(final Packet packet, final RemotingConnection conn) throws MessagingException
 {
 log.info("In interceptor, packet is " + packet.getType());
 if (packet.getType() == PacketImpl.PING)
 {
 log.info("Ignoring Ping packet.. it will be dropped");
 return false;
 }
 else
 {
 return true;
 }
 }
 };

 messagingService.getServer().getRemotingService().addInterceptor(noPongInterceptor);
 final int numberOfSessions = 30;
 final int numberOfThreads = 8;
 final CountDownLatch flagStart = new CountDownLatch(1);
 final CountDownLatch flagAligned = new CountDownLatch(numberOfThreads);

 final ClientSessionFactory csf = new ClientSessionFactoryImpl(transportConfig,
 null,
 DEFAULT_CONNECTION_LOAD_BALANCING_POLICY_CLASS_NAME,
 PING_INTERVAL,
 (long)(PING_INTERVAL * 1.5),
 DEFAULT_CALL_TIMEOUT,
 DEFAULT_CONSUMER_WINDOW_SIZE,
 DEFAULT_CONSUMER_MAX_RATE,
 DEFAULT_SEND_WINDOW_SIZE,
 DEFAULT_PRODUCER_MAX_RATE,
 DEFAULT_MIN_LARGE_MESSAGE_SIZE,
 DEFAULT_BLOCK_ON_ACKNOWLEDGE,
 DEFAULT_BLOCK_ON_NON_PERSISTENT_SEND,
 DEFAULT_BLOCK_ON_PERSISTENT_SEND,
 DEFAULT_AUTO_GROUP,
 DEFAULT_MAX_CONNECTIONS,
 DEFAULT_PRE_ACKNOWLEDGE,
 DEFAULT_ACK_BATCH_SIZE,
 DEFAULT_RETRY_INTERVAL,
 DEFAULT_RETRY_INTERVAL_MULTIPLIER,
 DEFAULT_MAX_RETRIES_BEFORE_FAILOVER,
 DEFAULT_MAX_RETRIES_AFTER_FAILOVER);

 class LocalThread extends Thread
 {
 Throwable failure;

 @Override
 public void run()
 {
 try
 {
 // Start all at once to make concurrency worst
 flagAligned.countDown();
 flagStart.await();
 for (int i = 0; i < numberOfSessions; i++)
 {

 ClientSession session = csf.createSession(false, false, false);

 Thread.sleep(PING_INTERVAL * (RandomUtil.randomPositiveInt() % 3));

 session.close();
 }
 }
 catch (Throwable e)
 {
 failure = e;
 }
 }
 };

 LocalThread threads[] = new LocalThread[numberOfThreads];

 for (int i = 0; i < numberOfThreads; i++)
 {
 threads = new LocalThread();
 threads.start();
 }

 flagAligned.await();
 flagStart.countDown();

 Throwable e = null;
 for (LocalThread t : threads)
 {
 t.join();
 if (t.failure != null)
 {
 e = t.failure;
 }
 }

 if (e != null)
 {
 throw new Exception("Test Failed", e);
 }

}


            • Actions
          • 2. Re: Another DeadLock, something around InVM
            clebert.suconic
            clebert.suconic Mar 4, 2009 3:13 PM (in response to clebert.suconic)

            a pastebin would be better on this case:

            http://pastebin.com/f2f0de30d

              • Actions
            • 3. Re: Another DeadLock, something around InVM
              clebert.suconic
              clebert.suconic Mar 4, 2009 6:25 PM (in response to clebert.suconic)

              I believe (unless I"m missing something here) this is being caused by the recent changes on shutting down the executors:

              I have found three threads waiting the same lock (00007fc50a611708):


              On the thread dump (1):

              - "Thread-1" waiting to the lock, before a call to failover, while the executor is being used.

              - Another thread named "LocalThread i = 10" is holding the lock and waiting to shutdown the executor.


              I would consider this a dead lock, even though the VM didn't report it so as it didn't explicity used two lock objects.


              I am adding PingStressTest into stressTest (which is the test i previously reported here with a few modifications). The test is currently disabled. Change getNumberOfIterations to something like 10 or 100 to replicate this hang.


              The test passed for me after reverting r5990.

              I will run it a few more times and report the results later.



              ___________________________________________________________________________
              *1 - Thread dump

              "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
 java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
 at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
 at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
 - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
 - locked <0x00007fc50a611708> (a java.lang.Object)
 - locked <0x00007fc50a6116f8> (a java.lang.Object)
 at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
 at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
 at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)


"Thread-1 (group:jbm-pinger-threads-896472140)" daemon prio=10 tid=0x00007fc500385c00 nid=0x6463 waiting for monitor entry [0x0000000045ece000..0x0000000045ecec00]
 java.lang.Thread.State: BLOCKED (on object monitor)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failover(ConnectionManagerImpl.java:493)
 - waiting to lock <0x00007fc50a611708> (a java.lang.Object)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionFailed(ConnectionManagerImpl.java:411)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:530)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger.run(RemotingConnectionImpl.java:1537)
 - locked <0x00007fc50a614508> (a org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 at java.lang.Thread.run(Thread.java:619)



"LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
 java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
 at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
 at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
 - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
 - locked <0x00007fc50a611708> (a java.lang.Object)
 - locked <0x00007fc50a6116f8> (a java.lang.Object)
 at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
 at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
 at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)




                • Actions
              • 4. Re: Another DeadLock, something around InVM
                clebert.suconic
                clebert.suconic Mar 4, 2009 6:39 PM (in response to clebert.suconic)

                Just had another look at the original thread dump (first one reported on this thread, that I got on hudson-build 3761):


                I can also see the waiting to shutdown the executor there:



                 [junit] "main" prio=1 tid=0x08deb950 nid=0x76c8 waiting on condition [0xffc33000..0xffc33d58]
 [junit] at sun.misc.Unsafe.park(Native Method)
 [junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
 [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
 [junit] at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1028)
 [junit] at org.jboss.messaging.integration.transports.netty.NettyAcceptor.stop(NettyAcceptor.java:354)
 [junit] - locked <0xb2718638> (a org.jboss.messaging.integration.transports.netty.NettyAcceptor)
 [junit] at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl.stop(RemotingServiceImpl.java:213)
 [junit] - locked <0xb2714c30> (a org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl)
 [junit] at org.jboss.messaging.core.server.impl.MessagingServiceImpl.stop(MessagingServiceImpl.java:67)
 [junit] at org.jboss.messaging.tests.integration.remoting.PingTest.tearDown(PingTest.java:102)
 [junit] at junit.framework.TestCase.runBare(TestCase.java:136)
 [junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
 [junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
 [junit] at junit.framework.TestResult.run(TestResult.java:109)
 [junit] at junit.framework.TestCase.run(TestCase.java:120)
 [junit] at junit.framework.TestSuite.runTest(TestSuite.java:230)
 [junit] at junit.framework.TestSuite.run(TestSuite.java:225)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)


                (Dump extracted from run 3761 at http://hudson.qa.jboss.com/hudson/view/JBM%202/job/JBM2-tests/3761/console)

                (Sorry non RedHatters for using an internal link.. but the relevant information is pasted here)



                So, this is the same condition replicated by PingStressTest.

                And I also confirmed the test passed without the changes from r5990.

                  • Actions
                • 5. Re: Another DeadLock, something around InVM
                  clebert.suconic
                  clebert.suconic Mar 4, 2009 10:45 PM (in response to clebert.suconic)

                  I was thinking now...


                  I guess the JORAM hang may be indirectly related to this:

                  My guess is, a missing ping happened after the server was stopped, while close was called at the same time. (What the test is about).

                  Maybe there is a way to avoid the close and missing ping at the same time. (Maybe making sure the Server is only stopped after the sessions are closed).


                  But this hang (replicated on PingStressTest) is probably the root cause.

                    • Actions
                  • 6. Re: Another DeadLock, something around InVM
                    timfox
                    timfox Mar 5, 2009 4:04 AM (in response to clebert.suconic)

                    Guys- please do not use pastebin links (internal or external) in forum threads.

                    Pastebins are a temporary resource, so if anyone comes back in days/weeks/months for now they won't be able to access the info.

                      • Actions
                    • 7. Re: Another DeadLock, something around InVM
                      ataylor
                      ataylor Mar 5, 2009 5:32 AM (in response to clebert.suconic)

                      I'm seeing similar hangs with the same thread dump. Whats weird is that the following line hangs:

                      if (workerExecutor.awaitTermination(1, TimeUnit.SECONDS))


                      I'm not sure why this line would hang!

                        • Actions
                      • 8. Re: Another DeadLock, something around InVM
                        ataylor
                        ataylor Mar 5, 2009 5:35 AM (in response to clebert.suconic)

                        actually its not hanging, I didn't notice it was in a loop

                          • Actions
                        • 9. Re: Another DeadLock, something around InVM
                          timfox
                          timfox Mar 5, 2009 5:54 AM (in response to clebert.suconic)

                           

                          "ataylor" wrote:
                          I'm seeing similar hangs with the same thread dump. Whats weird is that the following line hangs:


                          Is this after the fix I applied this morning?

                          If so, please post thread dump. This thread is confusing since two completely different thread dumps were posted, which are possibly different issues.

                            • Actions
                          • 10. Re: Another DeadLock, something around InVM
                            ataylor
                            ataylor Mar 5, 2009 6:03 AM (in response to clebert.suconic)

                            yes after your fix was applied. The second thread has the Channel.lock.

                            [junit] "New I/O client worker #318-1" daemon prio=10 tid=0x080be400 nid=0x640d waiting on condition [0x6ea5c000..0x6ea5cf40]
 [junit] java.lang.Thread.State: WAITING (parking)
 [junit] at sun.misc.Unsafe.park(Native Method)
 [junit] - parking to wait for <0xadf7e648> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 [junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 [junit] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
 [junit] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
 [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1459)
 [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:845)
 [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:492)
 [junit] - locked <0xadf7e400> (a java.lang.Object)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingBufferHandler.bufferReceived(ConnectionManagerImpl.java:943)
 [junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:62)
 [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
 [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
 [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
 [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
 [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
 [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
 [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
 [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
 [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
 [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
 [junit] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
 [junit] at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
 [junit] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
 [junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
 [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 [junit] at java.lang.Thread.run(Thread.java:619)
 [junit]

                            "main" prio=10 tid=0x0805cc00 nid=0x606e waiting on condition [0xb7d5d000..0xb7d5e208]
 [junit] java.lang.Thread.State: TIMED_WAITING (parking)
 [junit] at sun.misc.Unsafe.park(Native Method)
 [junit] - parking to wait for <0xadf7bed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 [junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
 [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
 [junit] at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1245)
 [junit] at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
 [junit] - locked <0xadf7b6c8> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
 [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:333)
 [junit] - locked <0xadf7afd8> (a java.lang.Object)
 [junit] at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:862)
 [junit] - locked <0xadf7a8d8> (a org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl)
 [junit] at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:574)
 [junit] at org.jboss.messaging.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:572)
 [junit] - locked <0xadf742b8> (a org.jboss.messaging.jms.client.JBossConnectionFactory)
 [junit] at org.jboss.messaging.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:408)
 [junit] at org.jboss.test.messaging.jms.SecurityTest.testLoginValidUserInvalidPassword(SecurityTest.java:136)
 [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 [junit] at java.lang.reflect.Method.invoke(Method.java:597)
 [junit] at junit.framework.TestCase.runTest(TestCase.java:164)
 [junit] at junit.framework.TestCase.runBare(TestCase.java:130)
 [junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
 [junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
 [junit] at junit.framework.TestResult.run(TestResult.java:109)
 [junit] at junit.framework.TestCase.run(TestCase.java:120)
 [junit] at junit.framework.TestSuite.runTest(TestSuite.java:230)
 [junit] at junit.framework.TestSuite.run(TestSuite.java:225)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
 [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)




                              • Actions
                            • 11. Re: Another DeadLock, something around InVM
                              timfox
                              timfox Mar 5, 2009 6:06 AM (in response to clebert.suconic)

                              Ok, that's a separate issue. The one you pasted if for Netty, the one I fixed is invm transport.

                              That's why I said this thread confusing - it's two separate issues.

                              Ok... onto fixing the second one.

                                • Actions
                              • 12. Re: Another DeadLock, something around InVM
                                timfox
                                timfox Mar 5, 2009 6:13 AM (in response to clebert.suconic)

                                Andy- What test is running when you see this deadlock?

                                  • Actions
                                • 13. Re: Another DeadLock, something around InVM
                                  ataylor
                                  ataylor Mar 5, 2009 6:14 AM (in response to clebert.suconic)

                                  heres the complete trace for completeness.

                                  [junit] 2009-03-05 10:16:55
[junit] Full thread dump Java HotSpot(TM) Server VM (11.0-b12 mixed mode):
[junit]
[junit] "New I/O client worker #318-1" daemon prio=10 tid=0x080be400 nid=0x640d waiting on condition [0x6ea5c000..0x6ea5cf40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0xadf7e648> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
[junit] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
[junit] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
[junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1459)
[junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:845)
[junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:492)
[junit] - locked <0xadf7e400> (a java.lang.Object)
[junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingBufferHandler.bufferReceived(ConnectionManagerImpl.java:943)
[junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:62)
[junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
[junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
[junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
[junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
[junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
[junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
[junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
[junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
[junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
[junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
[junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
[junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
[junit] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
[junit] at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
[junit] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
[junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-14 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x084c2800 nid=0x6405 waiting on condition [0x6ef32000..0x6ef33140]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-13 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x0849e400 nid=0x6404 waiting on condition [0x6ed4c000..0x6ed4cfc0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-12 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x0807e400 nid=0x6403 waiting on condition [0x6ee90000..0x6ee91040]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-11 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x08476400 nid=0x6402 waiting on condition [0x6ee3f000..0x6ee3fec0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-10 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x080bd400 nid=0x6401 waiting on condition [0x6ec59000..0x6ec59f40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-9 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x08278800 nid=0x63f9 waiting on condition [0x6ed9d000..0x6ed9dec0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-8 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x08513800 nid=0x63f8 waiting on condition [0x6ecfb000..0x6ecfbf40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-7 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x0810d000 nid=0x63f7 waiting on condition [0x6ecaa000..0x6ecaadc0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-6 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x08317000 nid=0x63f6 waiting on condition [0x6eafe000..0x6eafee40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-5 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x08452400 nid=0x63f5 waiting on condition [0x6f025000..0x6f0260c0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Timer-11" daemon prio=10 tid=0x6faad400 nid=0x63d2 in Object.wait() [0x6eee1000..0x6eee1fc0]
[junit] java.lang.Thread.State: TIMED_WAITING (on object monitor)
[junit] at java.lang.Object.wait(Native Method)
[junit] - waiting on <0x74fe6288> (a java.util.TaskQueue)
[junit] at java.util.TimerThread.mainLoop(Timer.java:509)
[junit] - locked <0x74fe6288> (a java.util.TaskQueue)
[junit] at java.util.TimerThread.run(Timer.java:462)
[junit]
[junit] "Thread-4 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x6faac400 nid=0x63d1 waiting on condition [0x6f25c000..0x6f25d040]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-3 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x6fad4000 nid=0x63d0 waiting on condition [0x6f6ad000..0x6f6adec0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-2 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x6fad3800 nid=0x63cf waiting on condition [0x6fbb8000..0x6fbb8f40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-1 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x6f5db000 nid=0x63ce waiting on condition [0x6ef83000..0x6ef83dc0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-0 (group:JBM-scheduled-threads-16900005)" daemon prio=10 tid=0x6fa79c00 nid=0x63cd waiting on condition [0x6f4ad000..0x6f4ade40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fd0c10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-0 (group:JBM-scheduled-threads-19170579)" daemon prio=10 tid=0x6fab0c00 nid=0x63ca waiting on condition [0x6fb67000..0x6fb680c0]
[junit] java.lang.Thread.State: TIMED_WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fe6b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Timer-10" daemon prio=10 tid=0x7059a000 nid=0x63c9 in Object.wait() [0x6f65c000..0x6f65d140]
[junit] java.lang.Thread.State: TIMED_WAITING (on object monitor)
[junit] at java.lang.Object.wait(Native Method)
[junit] - waiting on <0x74fdd530> (a java.util.TaskQueue)
[junit] at java.util.TimerThread.mainLoop(Timer.java:509)
[junit] - locked <0x74fdd530> (a java.util.TaskQueue)
[junit] at java.util.TimerThread.run(Timer.java:462)
[junit]
[junit] "New I/O server boss #6 (channelId: 29303475, /127.0.0.1:5445)" daemon prio=10 tid=0x6fab0000 nid=0x63c8 runnable [0x6f6fe000..0x6f6fefc0]
[junit] java.lang.Thread.State: RUNNABLE
[junit] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
[junit] - locked <0x74fda638> (a java.lang.Object)
[junit] at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:204)
[junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "pool-16-thread-1" prio=10 tid=0x6fa73400 nid=0x63c5 waiting on condition [0x6fe69000..0x6fe6a040]
[junit] java.lang.Thread.State: TIMED_WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74fe7238> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-4 (group:jbm-pinger-threads-12525344)" daemon prio=10 tid=0x6f52f400 nid=0x60a5 waiting on condition [0x6f169000..0x6f16a040]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74dac8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-3 (group:jbm-pinger-threads-12525344)" daemon prio=10 tid=0x6fc48800 nid=0x60a2 waiting on condition [0x6f1ba000..0x6f1badc0]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74dac8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-2 (group:jbm-pinger-threads-12525344)" daemon prio=10 tid=0x6fad2c00 nid=0x609f waiting on condition [0x6f20b000..0x6f20c140]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74dac8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-1 (group:jbm-pinger-threads-12525344)" daemon prio=10 tid=0x6f574000 nid=0x609b waiting on condition [0x6f2ad000..0x6f2adf40]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74dac8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Thread-0 (group:jbm-pinger-threads-12525344)" daemon prio=10 tid=0x70584c00 nid=0x6097 waiting on condition [0x6f45c000..0x6f45d140]
[junit] java.lang.Thread.State: WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0x74dac8d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
[junit] at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
[junit] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
[junit] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] "Low Memory Detector" daemon prio=10 tid=0x70592800 nid=0x6077 runnable [0x00000000..0x00000000]
[junit] java.lang.Thread.State: RUNNABLE
[junit]
[junit] "CompilerThread1" daemon prio=10 tid=0x70590400 nid=0x6076 waiting on condition [0x00000000..0x7018b378]
[junit] java.lang.Thread.State: RUNNABLE
[junit]
[junit] "CompilerThread0" daemon prio=10 tid=0x7058ec00 nid=0x6075 waiting on condition [0x00000000..0x7020c3f8]
[junit] java.lang.Thread.State: RUNNABLE
[junit]
[junit] "Signal Dispatcher" daemon prio=10 tid=0x7058d000 nid=0x6074 waiting on condition [0x00000000..0x00000000]
[junit] java.lang.Thread.State: RUNNABLE
[junit]
[junit] "Finalizer" daemon prio=10 tid=0x7057c400 nid=0x6073 in Object.wait() [0x70670000..0x70670e40]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(Native Method)
[junit] - waiting on <0x74c7cc08> (a java.lang.ref.ReferenceQueue$Lock)
[junit] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
[junit] - locked <0x74c7cc08> (a java.lang.ref.ReferenceQueue$Lock)
[junit] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
[junit] at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
[junit]
[junit] "Reference Handler" daemon prio=10 tid=0x7057b000 nid=0x6072 in Object.wait() [0x702ed000..0x702ee0c0]
[junit] java.lang.Thread.State: WAITING (on object monitor)
[junit] at java.lang.Object.wait(Native Method)
[junit] - waiting on <0x74c7cbe0> (a java.lang.ref.Reference$Lock)
[junit] at java.lang.Object.wait(Object.java:485)
[junit] at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
[junit] - locked <0x74c7cbe0> (a java.lang.ref.Reference$Lock)
[junit]
[junit] "main" prio=10 tid=0x0805cc00 nid=0x606e waiting on condition [0xb7d5d000..0xb7d5e208]
[junit] java.lang.Thread.State: TIMED_WAITING (parking)
[junit] at sun.misc.Unsafe.park(Native Method)
[junit] - parking to wait for <0xadf7bed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
[junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
[junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
[junit] at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1245)
[junit] at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
[junit] - locked <0xadf7b6c8> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
[junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
[junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
[junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:333)
[junit] - locked <0xadf7afd8> (a java.lang.Object)
[junit] at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:862)
[junit] - locked <0xadf7a8d8> (a org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl)
[junit] at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:574)
[junit] at org.jboss.messaging.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:572)
[junit] - locked <0xadf742b8> (a org.jboss.messaging.jms.client.JBossConnectionFactory)
[junit] at org.jboss.messaging.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:408)
[junit] at org.jboss.test.messaging.jms.SecurityTest.testLoginValidUserInvalidPassword(SecurityTest.java:136)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:164)
[junit] at junit.framework.TestCase.runBare(TestCase.java:130)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
[junit] at junit.framework.TestResult.run(TestResult.java:109)
[junit] at junit.framework.TestCase.run(TestCase.java:120)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:230)
[junit] at junit.framework.TestSuite.run(TestSuite.java:225)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
[junit]
[junit] "VM Thread" prio=10 tid=0x70577800 nid=0x6071 runnable
[junit]
[junit] "GC task thread#0 (ParallelGC)" prio=10 tid=0x08064000 nid=0x606f runnable
[junit]
[junit] "GC task thread#1 (ParallelGC)" prio=10 tid=0x08065400 nid=0x6070 runnable
[junit]
[junit] "VM Periodic Task Thread" prio=10 tid=0x70594400 nid=0x6078 waiting on condition
[junit]
[junit] JNI global references: 902
[junit]
[junit] Heap
[junit] PSYoungGen total 111616K, used 14022K [0xad9f0000, 0xb4bb0000, 0xb4bb0000)
[junit] eden space 107072K, 13% used [0xad9f0000,0xae7a19d8,0xb4280000)
[junit] from space 4544K, 0% used [0xb4710000,0xb4710000,0xb4b80000)
[junit] to space 4672K, 0% used [0xb4280000,0xb4280000,0xb4710000)
[junit] PSOldGen total 9920K, used 5738K [0x74bb0000, 0x75560000, 0xad9f0000)
[junit] object space 9920K, 57% used [0x74bb0000,0x7514a868,0x75560000)
[junit] PSPermGen total 28160K, used 15508K [0x70bb0000, 0x72730000, 0x74bb0000)
[junit] object space 28160K, 55% used [0x70bb0000,0x71ad51d8,0x72730000)
[junit]
 |
 |
 |


                                    • Actions
                                  • 14. Re: Another DeadLock, something around InVM
                                    clebert.suconic
                                    clebert.suconic Mar 5, 2009 8:22 AM (in response to clebert.suconic)

                                     

                                    "timfox" wrote:
                                    Guys- please do not use pastebin links (internal or external) in forum threads.

                                    Pastebins are a temporary resource, so if anyone comes back in days/weeks/months for now they won't be able to access the info.


                                    When I created that pastebin, I set it to be kept forever. You can say if it' s kept for 1 day, 1 month or forever. But I don' t know if that' s trustable.

                                      • Actions
                                    1 2 Previous Next