6 Replies Latest reply on Mar 7, 2009 10:26 AM by clebert.suconic

    JoramTests thread dumps...

    clebert.suconic

      I have added a Thread to read Joram output's after its initialization.

      This way, if the thread hangs again, we can get the thread dump with a kill -3.

      Like, it hang on hudson today. I had access to the box but I couldn't get much information as it didn't throw the output anywhere.

        • 1. Re: JoramTests thread dumps...
          clebert.suconic

          And it just produced this output:


          Full thread dump Java HotSpot(TM) Server VM (1.5.0_15-b04 mixed mode):
          
          "New I/O server worker #1-3" daemon prio=1 tid=0x8b8d2270 nid=0x5958 runnable [0x8a9cc000..0x8a9ccea0]
           at java.io.FileOutputStream.writeBytes(Native Method)
           at java.io.FileOutputStream.write(FileOutputStream.java:260)
           at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
           - locked <0x915485a8> (a java.io.BufferedOutputStream)
           at java.io.PrintStream.write(PrintStream.java:412)
           - locked <0x91523a40> (a java.io.PrintStream)
           at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
           at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
           at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
           at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
           - locked <0x915b4290> (a java.io.OutputStreamWriter)
           at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
           at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
           - locked <0x915b41f8> (a java.util.logging.ConsoleHandler)
           at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
           at java.util.logging.Logger.log(Logger.java:452)
           at java.util.logging.Logger.doLog(Logger.java:474)
           at java.util.logging.Logger.log(Logger.java:563)
           at org.jboss.messaging.core.logging.Logger.error(Logger.java:103)
           at org.jboss.messaging.core.filter.impl.FilterImpl.<init>(FilterImpl.java:124)
           at org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleCreateConsumer(ServerSessionImpl.java:1198)
           at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCreateConsumer(ServerSessionImpl.java:360)
           at org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:120)
           at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1469)
           at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:845)
           at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:492)
           - locked <0x915b9640> (a java.lang.Object)
           at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
           at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:73)
           at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
           at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
           at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
           at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
           at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
           at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
           at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
           at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
           at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
           at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
           at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
           at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
           at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
           at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
           at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
           at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
           at java.lang.Thread.run(Thread.java:595)
          
          "RMI LeaseChecker" daemon prio=1 tid=0x083b65a8 nid=0x5953 waiting on condition [0x8ab1a000..0x8ab1afa0]
           at java.lang.Thread.sleep(Native Method)
           at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
           at java.lang.Thread.run(Thread.java:595)
          
          "Timer-1" daemon prio=1 tid=0x8c3d5330 nid=0x5948 in Object.wait() [0x8aefe000..0x8aefefa0]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x915b6c18> (a java.util.TaskQueue)
           at java.util.TimerThread.mainLoop(Timer.java:509)
           - locked <0x915b6c18> (a java.util.TaskQueue)
           at java.util.TimerThread.run(Timer.java:462)
          
          "Thread-0 (group:JBM-scheduled-threads-28336193)" daemon prio=1 tid=0x8c3cf3e8 nid=0x5947 waiting on condition [0x8b08d000..0x8b08e020]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:135)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "NamingBootstrap Pool(1)-1" daemon prio=1 tid=0x08193848 nid=0x5944 runnable [0x8b115000..0x8b1160a0]
           at java.net.PlainSocketImpl.socketAccept(Native Method)
           at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           - locked <0x915a22d8> (a java.net.SocksSocketImpl)
           at java.net.ServerSocket.implAccept(ServerSocket.java:450)
           at java.net.ServerSocket.accept(ServerSocket.java:421)
           at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
           at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
           at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
           at java.lang.Thread.run(Thread.java:595)
          
          "GC Daemon" daemon prio=1 tid=0x0827e670 nid=0x5943 in Object.wait() [0x8b197000..0x8b197120]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x91548310> (a sun.misc.GC$LatencyLock)
           at sun.misc.GC$Daemon.run(GC.java:100)
           - locked <0x91548310> (a sun.misc.GC$LatencyLock)
          
          "RMI Reaper" prio=1 tid=0x0812bc30 nid=0x5942 in Object.wait() [0x8b217000..0x8b217da0]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x91548318> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
           - locked <0x91548318> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
           at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
           at java.lang.Thread.run(Thread.java:595)
          
          "Timer-0" daemon prio=1 tid=0x0812b580 nid=0x5941 in Object.wait() [0x8b298000..0x8b298e20]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x91520508> (a java.util.TaskQueue)
           at java.lang.Object.wait(Object.java:474)
           at java.util.TimerThread.mainLoop(Timer.java:483)
           - locked <0x91520508> (a java.util.TaskQueue)
           at java.util.TimerThread.run(Timer.java:462)
          
          "RMI TCP Accept-1098" daemon prio=1 tid=0x08280998 nid=0x5940 runnable [0x8b319000..0x8b319ea0]
           at java.net.PlainSocketImpl.socketAccept(Native Method)
           at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           - locked <0x91547e70> (a java.net.SocksSocketImpl)
           at java.net.ServerSocket.implAccept(ServerSocket.java:450)
           at java.net.ServerSocket.accept(ServerSocket.java:421)
           at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
           at java.lang.Thread.run(Thread.java:595)
          
          "Low Memory Detector" daemon prio=1 tid=0x8cfa7f30 nid=0x593c runnable [0x00000000..0x00000000]
          
          "CompilerThread1" daemon prio=1 tid=0x8cfa6b50 nid=0x593b waiting on condition [0x00000000..0x8ca79248]
          
          "CompilerThread0" daemon prio=1 tid=0x8cfa5bd0 nid=0x593a waiting on condition [0x00000000..0x8cafa2c8]
          
          "AdapterThread" daemon prio=1 tid=0x8cfa4c20 nid=0x5939 waiting on condition [0x00000000..0x00000000]
          
          "Signal Dispatcher" daemon prio=1 tid=0x8cfa3de8 nid=0x5938 waiting on condition [0x00000000..0x00000000]
          
          "Finalizer" daemon prio=1 tid=0x8cf9a840 nid=0x5937 in Object.wait() [0x8ce7d000..0x8ce7de20]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x915485a0> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
           - locked <0x915485a0> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
           at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
          
          "Reference Handler" daemon prio=1 tid=0x8cf9a310 nid=0x5936 in Object.wait() [0x8cefe000..0x8cefeea0]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x91523a90> (a java.lang.ref.Reference$Lock)
           at java.lang.Object.wait(Object.java:474)
           at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
           - locked <0x91523a90> (a java.lang.ref.Reference$Lock)
          
          "main" prio=1 tid=0x08062408 nid=0x5930 waiting on condition [0xbfffb000..0xbfffb5a8]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
           at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1028)
           at org.jboss.netty.util.ExecutorUtil.terminate(ExecutorUtil.java:82)
           at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:151)
           at org.jboss.messaging.integration.transports.netty.NettyAcceptor.stop(NettyAcceptor.java:329)
           - locked <0x915bd7f0> (a org.jboss.messaging.integration.transports.netty.NettyAcceptor)
           at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl.stop(RemotingServiceImpl.java:213)
           - locked <0x915b8260> (a org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl)
           at org.jboss.messaging.core.server.impl.MessagingServiceImpl.stop(MessagingServiceImpl.java:67)
           at org.jboss.test.jms.SpawnedJMSServer.main(SpawnedJMSServer.java:100)
          
          "VM Thread" prio=1 tid=0x8cf97f00 nid=0x5935 runnable
          
          "GC task thread#0 (ParallelGC)" prio=1 tid=0x0807c030 nid=0x5931 runnable
          
          "GC task thread#1 (ParallelGC)" prio=1 tid=0x0807d088 nid=0x5932 runnable
          
          "GC task thread#2 (ParallelGC)" prio=1 tid=0x0807dca0 nid=0x5933 runnable
          
          "GC task thread#3 (ParallelGC)" prio=1 tid=0x0807e8c0 nid=0x5934 runnable
          
          "VM Periodic Task Thread" prio=1 tid=0x8cfa9ad8 nid=0x593d waiting on condition
          



          The Testsuite Thread dump is here for completeness, but it doesn't show anything interesting:

          Full thread dump Java HotSpot(TM) Server VM (1.5.0_15-b04 mixed mode):
          
          "Thread-131" prio=1 tid=0x6bf1c988 nid=0x594b runnable [0x6a42f000..0x6a42ffa0]
           at java.io.FileInputStream.readBytes(Native Method)
           at java.io.FileInputStream.read(FileInputStream.java:194)
           at java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
           at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
           - locked <0x716a3f50> (a java.io.BufferedInputStream)
           at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
           at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
           at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
           - locked <0x716a3fc8> (a java.io.InputStreamReader)
           at java.io.InputStreamReader.read(InputStreamReader.java:167)
           at java.io.BufferedReader.fill(BufferedReader.java:136)
           at java.io.BufferedReader.readLine(BufferedReader.java:299)
           - locked <0x716a3fc8> (a java.io.InputStreamReader)
           at java.io.BufferedReader.readLine(BufferedReader.java:362)
           at org.jboss.test.jms.JBossMessagingAdmin$1.run(JBossMessagingAdmin.java:284)
          
          "process reaper" daemon prio=1 tid=0x6baf9540 nid=0x592f runnable [0x6b71e000..0x6b71e120]
           at java.lang.UNIXProcess.waitForProcessExit(Native Method)
           at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
           at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
          
          "Thread-4 (group:jbm-pinger-threads-11508030)" daemon prio=1 tid=0x083d4b48 nid=0x4138 waiting on condition [0x6a6b4000..0x6a6b4fa0]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:131)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "Thread-3 (group:jbm-pinger-threads-11508030)" daemon prio=1 tid=0x083e0e78 nid=0x4114 waiting on condition [0x6a32d000..0x6a32df20]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:131)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "Thread-2 (group:jbm-pinger-threads-11508030)" daemon prio=1 tid=0x083dcf00 nid=0x4110 waiting on condition [0x6a4b0000..0x6a4b0fa0]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:131)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "Thread-1 (group:jbm-pinger-threads-11508030)" daemon prio=1 tid=0x083d56c8 nid=0x410c waiting on condition [0x6a633000..0x6a633e20]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:131)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "GC Daemon" daemon prio=1 tid=0x083caef0 nid=0x4105 in Object.wait() [0x6a88b000..0x6a88c020]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x7162aac0> (a sun.misc.GC$LatencyLock)
           at sun.misc.GC$Daemon.run(GC.java:100)
           - locked <0x7162aac0> (a sun.misc.GC$LatencyLock)
          
          "RMI RenewClean-[localhost:1098]" daemon prio=1 tid=0x083ac4f8 nid=0x4104 in Object.wait() [0x6a90c000..0x6a90cfa0]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x7161eda0> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
           - locked <0x7161eda0> (a java.lang.ref.ReferenceQueue$Lock)
           at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
           at java.lang.Thread.run(Thread.java:595)
          
          "Thread-0 (group:jbm-pinger-threads-11508030)" daemon prio=1 tid=0x083abc10 nid=0x40ff waiting on condition [0x6adfe000..0x6adff120]
           at sun.misc.Unsafe.park(Native Method)
           at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
           at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
           at java.util.concurrent.DelayQueue.take(DelayQueue.java:131)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:504)
           at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:497)
           at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)
           at java.lang.Thread.run(Thread.java:595)
          
          "Low Memory Detector" daemon prio=1 tid=0x6ce02880 nid=0x40da runnable [0x00000000..0x00000000]
          
          "CompilerThread1" daemon prio=1 tid=0x6ce014a0 nid=0x40d9 waiting on condition [0x00000000..0x6c8f81c8]
          
          "CompilerThread0" daemon prio=1 tid=0x6ce00520 nid=0x40d8 waiting on condition [0x00000000..0x6c979348]
          
          "AdapterThread" daemon prio=1 tid=0x08125b48 nid=0x40d7 waiting on condition [0x00000000..0x00000000]
          
          "Signal Dispatcher" daemon prio=1 tid=0x08124c70 nid=0x40d6 waiting on condition [0x00000000..0x00000000]
          
          "Finalizer" daemon prio=1 tid=0x0811b570 nid=0x40d5 in Object.wait() [0x6ccfc000..0x6ccfcda0]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x7154dc60> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
           - locked <0x7154dc60> (a java.lang.ref.ReferenceQueue$Lock)
           at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
           at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
          
          "Reference Handler" daemon prio=1 tid=0x08119f00 nid=0x40d4 in Object.wait() [0x6cd7d000..0x6cd7df20]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x7160e2d0> (a java.lang.ref.Reference$Lock)
           at java.lang.Object.wait(Object.java:474)
           at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
           - locked <0x7160e2d0> (a java.lang.ref.Reference$Lock)
          
          "main" prio=1 tid=0x08062d98 nid=0x40c9 in Object.wait() [0xbfffa000..0xbfffb188]
           at java.lang.Object.wait(Native Method)
           - waiting on <0x716a4078> (a java.lang.UNIXProcess)
           at java.lang.Object.wait(Object.java:474)
           at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)
           - locked <0x716a4078> (a java.lang.UNIXProcess)
           at org.jboss.test.jms.JBossMessagingAdmin.stopServer(JBossMessagingAdmin.java:311)
           at org.objectweb.jtests.jms.framework.JMSTestCase.tearDown(JMSTestCase.java:116)
           at org.objectweb.jtests.jms.framework.PTPTestCase.tearDown(PTPTestCase.java:182)
           at junit.framework.TestCase.runBare(TestCase.java:136)
           at junit.framework.TestResult$1.protect(TestResult.java:106)
           at junit.framework.TestResult.runProtected(TestResult.java:124)
           at junit.framework.TestResult.run(TestResult.java:109)
           at junit.framework.TestCase.run(TestCase.java:120)
           at junit.framework.TestSuite.runTest(TestSuite.java:230)
           at junit.framework.TestSuite.run(TestSuite.java:225)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
           at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
          
          "VM Thread" prio=1 tid=0x081179b8 nid=0x40d3 runnable
          
          "GC task thread#0 (ParallelGC)" prio=1 tid=0x0807c9d8 nid=0x40cd runnable
          
          "GC task thread#1 (ParallelGC)" prio=1 tid=0x0807da30 nid=0x40ce runnable
          
          "GC task thread#2 (ParallelGC)" prio=1 tid=0x0807e648 nid=0x40cf runnable
          
          "GC task thread#3 (ParallelGC)" prio=1 tid=0x0807f268 nid=0x40d0 runnable
          
          "VM Periodic Task Thread" prio=1 tid=0x6ce04428 nid=0x40db waiting on condition
          
          


          • 2. Re: JoramTests thread dumps...
            timfox

            LOL, so it's blocked writing logging output to stdout, which isn't being read on the other side.

            That explains it. :)

            • 3. Re: JoramTests thread dumps...
              clebert.suconic


              I am capturing the output.. that how I actually captured the server's thread dump.

              But I believe the log output goes to System.err, right? that's probably it.

              • 4. Re: JoramTests thread dumps...
                clebert.suconic

                I added a loop on start with a bunch of System.err.println and System.err.flush, and it hang, while it passed after reading the errorStream.

                I will commit the reader.. Hopefully it won't hang any more.

                • 5. Re: JoramTests thread dumps...
                  timfox

                  I don't know what happened to your commit, but the version I got from svn looks like this:

                  public void startServer() throws Exception
                   {
                   serverProcess = SpawnedVMSupport.spawnVM(SpawnedJMSServer.class.getName(), false);
                   InputStreamReader isr = new InputStreamReader(serverProcess.getInputStream());
                  
                   final BufferedReader br = new BufferedReader(isr);
                   String line = null;
                   while ((line = br.readLine()) != null)
                   {
                   System.out.println(line);
                   line.replace('|', '\n');
                   if (line.startsWith("Listening for transport"))
                   {
                   continue;
                   }
                   else if ("OK".equals(line.trim()))
                   {
                   new Thread()
                   {
                   public void run()
                   {
                   try
                   {
                   String line = null;
                   while ((line = br.readLine()) != null)
                   {
                   System.out.println("JoramServerOutput: " + line);
                   }
                   }
                   catch (Exception e)
                   {
                   e.printStackTrace();
                   }
                   }
                   }.start();
                   return;
                   }
                   else
                   {
                   // something went wrong with the server, destroy it:
                   serverProcess.destroy();
                   throw new IllegalStateException("Unable to start the spawned server :" + line);
                   }
                   }
                   }
                  


                  You'd need to read stderr on a different thread to get it to work, also don't exit without joining the threads.

                  • 6. Re: JoramTests thread dumps...
                    clebert.suconic

                    I moved the thread to SpawnVMSupport.


                    The thread will exit when readLine == null.