10 Replies Latest reply on Mar 29, 2004 1:11 PM by Richard Bielak

    JMS hang/crash (Jboss 3.2.3)

    Richard Bielak Newbie

      We had a JMS crash/hang occur in our development JBoss 3.2.3. An OutOfMemory error occured first, and then the clients got into a weird state. Some clients got stuck while trying to reconnect to Jboss in InitialContext look up.

      We had a program that was monitoring topic message counts and it started to report problems few minutes before.

      Here is first incation of problem in the log:

      2004-03-17 10:29:01,635 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.AddMsg791646028[msgTy
      pe: m_addMessage, msgID: 430942, error: null]
      java.lang.OutOfMemoryError
      2004-03-17 10:29:01,635 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.AddMsg274059084[msgTy
      pe: m_addMessage, msgID: 1163451, error: null]
      java.io.IOException: Client is not connected
      at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java(Compiled Code))
      at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java(Inlined Compiled Code))
      at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java(Compiled Code))
      at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java(Compiled Code))
      at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java(Compiled Code))
      at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java(Compiled Code))
      at java.lang.Thread.run(Thread.java:568)
      2004-03-17 10:45:19,321 WARN [com.javtech.tradescope] JMS service not reachable ... will try again in a few seconds
      2004-03-17 10:42:25,503 ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] Endpoint ServerSocket[addr=/0.0.0.0,port=0,localport=8081
      ] ignored exception: java.net.SocketException: Connection reset by peer

      And then we see lots of these:

      2004-03-17 11:10:34,387 ERROR [STDERR] java.lang.OutOfMemoryError:
      2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:68)
      2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
      2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenera
      tor.java:111)
      2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:317)
      2004-03-17 11:10:34,387 ERROR [STDERR] at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java(Compiled Code
      ))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.access$1100(ObjectStreamClass.java:72)
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:399)
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method)
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.(ObjectStreamClass.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java(Inlined Compiled Co
      de))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java(Compiled Code))
      2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java(Compiled Code))
      2004-03-17 11:10:34,389 ERROR [STDERR] at org.jnp.server.Main.run(Main.java:306)
      2004-03-17 11:10:34,389 ERROR [STDERR] at java.lang.Thread.run(Thread.java(Compiled Code))
      2004-03-17 11:10:34,389 ERROR [STDERR] java.lang.OutOfMemoryError:


      I took thread dump while the server was in this state. It was from IBM's JVM running on Linux. Please let me know where to post it.

      ...richie









        • 1. Re: JMS hang/crash (Jboss 3.2.3)
          Richard Bielak Newbie

          I forgot to add that when I tried to shut Jboss down with shutdown.sh
          script it did not work. The script timed out.

          ..richie

          • 2. Re: JMS hang/crash (Jboss 3.2.3)
            Richard Bielak Newbie

            Here is the extract ot the threaddump at the time JMS was stuck:


            2XMFULLTHDDUMP Full thread dump Classic VM (J2RE 1.4.1 IBM build cxia32141-20030522, native threads):
            3XMTHREADINFO "JNP Server" (TID:0x1051DE78, sys_thread_t:0x8C2E2F0, state:R, native ID:0xBCA4000F) prio=5
            4XESTACKTRACE at java.lang.Thread.start(Native Method)
            4XESTACKTRACE at org.jnp.server.Main.listen(Main.java(Compiled Code))
            4XESTACKTRACE at org.jnp.server.Main.run(Main.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 00000000, EBX : 08C2E898, ECX : 08C2E8BC
            3HPREGVALUES EDX : 00000000, ESI : 08C2E8BC, EDI : 00000000
            3HPREGVALUES EBP : BDDFF70C, ESP : BDDFF6C0, EIP : 4012D9F7
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "JNP Server" PID 21907
            NULL -------------------------
            3HPSTACKLINE __sched_yield at 4012D9F7 in libc.so.6
            3HPSTACKLINE xeThreadCreate at 4034F2B1 in libjvm.so
            3HPSTACKLINE xmCreateThread at 40353E06 in libjvm.so
            3HPSTACKLINE JVM_StartThread at 4025FBD4 in libjvm.so
            3HPSTACKLINE 40BD608E
            3HPSTACKLINE 0
            NULL
            3XMTHREADINFO "MessageListenerThread - ts.toserver.engineinfo" (TID:0x1050F300, sys_thread_t:0x8F45178, state:CW, native ID:0xBC95C022) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08F45464, EBX : 08F45178, ECX : 403DAE00
            3HPREGVALUES EDX : 08F45464, ESI : FFFFFFFF, EDI : 08ED6710
            3HPREGVALUES EBP : BB7FF750, ESP : BB7FF718, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.toserver.engineinfo" PID 21850
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "MessageListenerThread - ts.toserver.masterconfig" (TID:0x1050F410, sys_thread_t:0x8F51FD0, state:CW, native ID:0xBC954020) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08F522BC, EBX : 08F51FD0, ECX : 403DAE00
            3HPREGVALUES EDX : 08F522BC, ESI : FFFFFFFF, EDI : 08EDF530
            3HPREGVALUES EBP : BBBFF750, ESP : BBBFF718, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.toserver.masterconfig" PID 21848
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "MessageListenerThread - ts.agent.heartbeat" (TID:0x1050EA50, sys_thread_t:0x8F44BD0, state:CW, native ID:0xBC94C01F) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08F44EBC, EBX : 08F44BD0, ECX : 403DAE00
            3HPREGVALUES EDX : 08F44EBC, ESI : FFFFFFFF, EDI : 08EE5A10
            3HPREGVALUES EBP : BBDFF750, ESP : BBDFF718, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.agent.heartbeat" PID 21846
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "MessageListenerThread - ts.fromserver.alert" (TID:0x1050EB60, sys_thread_t:0x875B888, state:CW, native ID:0xBC944007) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 0875BB74, EBX : 0875B888, ECX : 403DAE00
            3HPREGVALUES EDX : 0875BB74, ESI : FFFFFFFF, EDI : 08EE3A90
            3HPREGVALUES EBP : BEDFF750, ESP : BEDFF718, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.fromserver.alert" PID 21844
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "MessageListenerThread - ts.agent.command" (TID:0x1050EC70, sys_thread_t:0x92B1188, state:CW, native ID:0xBC93C005) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 092B1474, EBX : 092B1188, ECX : 403DAE00
            3HPREGVALUES EDX : 092B1474, ESI : FFFFFFFF, EDI : 08EBAA48
            3HPREGVALUES EBP : BF1FF750, ESP : BF1FF718, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.agent.command" PID 21842
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "RMI ConnectionExpiration-[192.168.140.22:1098]" (TID:0x102B4008, sys_thread_t:0x9186A90, state:R, native ID:0x7758C014) prio=5
            4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
            4XESTACKTRACE at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 09186D7C, EBX : 09186A90, ECX : 00000001
            3HPREGVALUES EDX : 09186D7C, ESI : 00003A98, EDI : 0805CFE8
            3HPREGVALUES EBP : BD3FF78C, ESP : BD3FF754, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "RMI ConnectionExpiration-[192.168.140.22:1098]" PID 12464
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
            NULL
            3XMTHREADINFO "RMI TCP Connection(21636)-192.168.140.22" (TID:0x102B40B8, sys_thread_t:0x42C01988, state:R, native ID:0x7758400E) prio=5
            4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
            4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
            4XESTACKTRACE at java.io.BufferedInputStream.fill(BufferedInputStream.java(Compiled Code))
            4XESTACKTRACE at java.io.BufferedInputStream.read(BufferedInputStream.java(Compiled Code))
            4XESTACKTRACE at java.io.FilterInputStream.read(FilterInputStream.java(Compiled Code))
            4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java(Compiled Code))
            4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : FFFFFFFC, EBX : BDFFF4C8, ECX : 00000001
            3HPREGVALUES EDX : 006DDD00, ESI : 0830DEC0, EDI : BDFFF4C8
            3HPREGVALUES EBP : BDFFF450, ESP : BDFFF444, EIP : 401392C0
            3HPREGVALUES EFLAGS : 00000212
            NULL
            3HPNATIVESTACK Native Stack of "RMI TCP Connection(21636)-192.168.140.22" PID 12462
            NULL -------------------------
            3HPSTACKLINE __poll at 401392C0 in libc.so.6
            3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
            3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so

            3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
            3HPSTACKLINE 40F415BF
            NULL
            3XMTHREADINFO "Thread-64290" (TID:0x113C2F08, sys_thread_t:0x43316370, state:CW, native ID:0x536F80C2) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4331665C, EBX : 43316370, ECX : 43316370
            3HPREGVALUES EDX : 4331665C, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A77FF648, ESP : A77FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64290" PID 6127
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64289" (TID:0x113C3010, sys_thread_t:0x43315DC8, state:CW, native ID:0x536F00C0) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 433160B4, EBX : 43315DC8, ECX : 43315DC8
            3HPREGVALUES EDX : 433160B4, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A7BFF648, ESP : A7BFF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64289" PID 6121
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64288" (TID:0x113C3068, sys_thread_t:0x43315820, state:R, native ID:0x536EC0BF) prio=5
            4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:193)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 43315B0C, EBX : 43315820, ECX : 00000001
            3HPREGVALUES EDX : 43315B0C, ESI : 00002710, EDI : 0805CFE8
            3HPREGVALUES EBP : A7DFF5E0, ESP : A7DFF5A8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64288" PID 6120
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
            NULL
            3XMTHREADINFO "Thread-64287" (TID:0x113C31C8, sys_thread_t:0x43313308, state:CW, native ID:0x536E00BC) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 433135F4, EBX : 43313308, ECX : 43313308
            3HPREGVALUES EDX : 433135F4, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A83FF648, ESP : A83FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64287" PID 6117
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64286" (TID:0x113C32D0, sys_thread_t:0x43312D60, state:CW, native ID:0x536D40BA) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4331304C, EBX : 43312D60, ECX : 43312D60
            3HPREGVALUES EDX : 4331304C, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A87FF648, ESP : A87FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64286" PID 6114
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64285" (TID:0x113C3328, sys_thread_t:0x43307168, state:CW, native ID:0x536CC0A6) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 43307454, EBX : 43307168, ECX : 43307168
            3HPREGVALUES EDX : 43307454, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : AAFFF648, ESP : AAFFF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64285" PID 5884
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64284" (TID:0x113C3430, sys_thread_t:0x433127B8, state:CW, native ID:0x536C00B8) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 43312AA4, EBX : 433127B8, ECX : 433127B8
            3HPREGVALUES EDX : 43312AA4, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A8BFF648, ESP : A8BFF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64284" PID 5881
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64280" (TID:0x106983E8, sys_thread_t:0x43312210, state:CW, native ID:0x536A80B6) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 433124FC, EBX : 43312210, ECX : 43312210
            3HPREGVALUES EDX : 433124FC, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A8FFF648, ESP : A8FFF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64280" PID 5871
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64279" (TID:0x106984F0, sys_thread_t:0x43307CB8, state:CW, native ID:0x5369C0B4) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 43307FA4, EBX : 43307CB8, ECX : 43307CB8
            3HPREGVALUES EDX : 43307FA4, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A93FF648, ESP : A93FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64279" PID 5865
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64278" (TID:0x106985F8, sys_thread_t:0x4330B730, state:CW, native ID:0x536900B1) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4330BA1C, EBX : 4330B730, ECX : 4330B730
            3HPREGVALUES EDX : 4330BA1C, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : A99FF648, ESP : A99FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64278" PID 5862
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64277" (TID:0x10698700, sys_thread_t:0x4330DC48, state:CW, native ID:0x53684071) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4330DF34, EBX : 4330DC48, ECX : 4330DC48
            3HPREGVALUES EDX : 4330DF34, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : B19FF648, ESP : B19FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64277" PID 5859
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64274" (TID:0x10293CB8, sys_thread_t:0x4330E1F0, state:CW, native ID:0x536600A0) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4330E4DC, EBX : 4330E1F0, ECX : 4330E1F0
            3HPREGVALUES EDX : 4330E4DC, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : ABBFF648, ESP : ABBFF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64274" PID 5850
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-64272" (TID:0x1066E7D8, sys_thread_t:0x4330C6E8, state:CW, native ID:0x53654082) prio=5
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
            4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4330C9D4, EBX : 4330C6E8, ECX : 4330C6E8
            3HPREGVALUES EDX : 4330C9D4, ESI : 101BCB68, EDI : 08EE4B10
            3HPREGVALUES EBP : AF7FF648, ESP : AF7FF610, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-64272" PID 5847
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor16" (TID:0x1032B228, sys_thread_t:0x8FE93B0, state:CW, native ID:0x7B6008A) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08FE969C, EBX : 08FE93B0, ECX : 403DAE00
            3HPREGVALUES EDX : 08FE969C, ESI : FFFFFFFF, EDI : 08D37070
            3HPREGVALUES EBP : AE7FF630, ESP : AE7FF5F8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor16" PID 2858
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor15" (TID:0x1032B298, sys_thread_t:0x8FE8E08, state:CW, native ID:0x7B5C089) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08FE90F4, EBX : 08FE8E08, ECX : 403DAE00
            3HPREGVALUES EDX : 08FE90F4, ESI : FFFFFFFF, EDI : 08E8B190
            3HPREGVALUES EBP : AE9FF630, ESP : AE9FF5F8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor15" PID 2857
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor14" (TID:0x1032B308, sys_thread_t:0x8FE52E0, state:CW, native ID:0x7B58088) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08FE55CC, EBX : 08FE52E0, ECX : 403DAE00
            3HPREGVALUES EDX : 08FE55CC, ESI : FFFFFFFF, EDI : 08EA0440
            3HPREGVALUES EBP : AEBFF630, ESP : AEBFF5F8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor14" PID 2856
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor13" (TID:0x1032B378, sys_thread_t:0x8FE4D38, state:CW, native ID:0x7B54086) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08FE5024, EBX : 08FE4D38, ECX : 403DAE00
            3HPREGVALUES EDX : 08FE5024, ESI : FFFFFFFF, EDI : 08EDBC30
            3HPREGVALUES EBP : AEFFF630, ESP : AEFFF5F8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor13" PID 2855
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-10" (TID:0x10207A70, sys_thread_t:0x8DC0218, state:CW, native ID:0x300073) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08DC0504, EBX : 08DC0218, ECX : 403DAE00
            3HPREGVALUES EDX : 08DC0504, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B15FF720, ESP : B15FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-10" PID 15867
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-9" (TID:0x101F9C88, sys_thread_t:0x8CFB890, state:CW, native ID:0x2C4056) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08CFBB7C, EBX : 08CFB890, ECX : 403DAE00
            3HPREGVALUES EDX : 08CFBB7C, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B4FFF720, ESP : B4FFF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-9" PID 15820
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-8" (TID:0x101A3340, sys_thread_t:0x8E848C0, state:CW, native ID:0x2B0053) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08E84BAC, EBX : 08E848C0, ECX : 403DAE00
            3HPREGVALUES EDX : 08E84BAC, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B55FF720, ESP : B55FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-8" PID 15812
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-7" (TID:0x101FBD00, sys_thread_t:0x8E715E0, state:CW, native ID:0x28C03B) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08E718CC, EBX : 08E715E0, ECX : 403DAE00
            3HPREGVALUES EDX : 08E718CC, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B85FF720, ESP : B85FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-7" PID 15776
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-6" (TID:0x10106C18, sys_thread_t:0x8C9EE48, state:CW, native ID:0x25C035) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08C9F134, EBX : 08C9EE48, ECX : 403DAE00
            3HPREGVALUES EDX : 08C9F134, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B91FF720, ESP : B91FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-6" PID 15764
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-131" (TID:0x100603D0, sys_thread_t:0x42025320, state:CW, native ID:0x24C023) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java:438)
            4XESTACKTRACE at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 4202560C, EBX : 42025320, ECX : 403DAE00
            3HPREGVALUES EDX : 4202560C, ESI : FFFFFFFF, EDI : 0805B9C0
            3HPREGVALUES EBP : BB5FF720, ESP : BB5FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-131" PID 15760
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-5" (TID:0x101B3948, sys_thread_t:0x8DE0970, state:CW, native ID:0x23006D) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08DE0C5C, EBX : 08DE0970, ECX : 403DAE00
            3HPREGVALUES EDX : 08DE0C5C, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B21FF720, ESP : B21FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-5" PID 15752
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor12" (TID:0x10222B28, sys_thread_t:0x8DA1A90, state:R, native ID:0x1AC02F) prio=5
            4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
            4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : FFFFFFFC, EBX : B9DFF428, ECX : 00000001
            3HPREGVALUES EDX : 00004E20, ESI : 0830DC20, EDI : B9DFF428
            3HPREGVALUES EBP : B9DFF3B0, ESP : B9DFF3A4, EIP : 401392C0
            3HPREGVALUES EFLAGS : 00000212
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor12" PID 15717
            NULL -------------------------
            3HPSTACKLINE __poll at 401392C0 in libc.so.6
            3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
            3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so
            3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
            3HPSTACKLINE 40F415BF
            NULL
            3XMTHREADINFO "TP-Processor11" (TID:0x10222C48, sys_thread_t:0x8713378, state:CW, native ID:0x1A002A) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08713664, EBX : 08713378, ECX : 403DAE00
            3HPREGVALUES EDX : 08713664, ESI : FFFFFFFF, EDI : 08C985A0
            3HPREGVALUES EBP : BA7FF608, ESP : BA7FF5D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor11" PID 15714
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor10" (TID:0x10222D10, sys_thread_t:0x8C5A478, state:CW, native ID:0x19803A) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08C5A764, EBX : 08C5A478, ECX : 403DAE00
            3HPREGVALUES EDX : 08C5A764, ESI : FFFFFFFF, EDI : 08D36A70
            3HPREGVALUES EBP : B87FF608, ESP : B87FF5D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor10" PID 15712
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor9" (TID:0x10222DD8, sys_thread_t:0x8D7B960, state:CW, native ID:0x19000D) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08D7BC4C, EBX : 08D7B960, ECX : 403DAE00
            3HPREGVALUES EDX : 08D7BC4C, ESI : FFFFFFFF, EDI : 08D36A10
            3HPREGVALUES EBP : BE1FF608, ESP : BE1FF5D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor9" PID 15710
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-4" (TID:0x101FAB70, sys_thread_t:0x8CFD4E0, state:CW, native ID:0x17C055) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08CFD7CC, EBX : 08CFD4E0, ECX : 403DAE00
            3HPREGVALUES EDX : 08CFD7CC, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B51FF720, ESP : B51FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-4" PID 15704
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-3" (TID:0x101FAC40, sys_thread_t:0x8CFBF80, state:CW, native ID:0x178054) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08CFC26C, EBX : 08CFBF80, ECX : 403DAE00
            3HPREGVALUES EDX : 08CFC26C, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B53FF720, ESP : B53FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-3" PID 15703
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-63" (TID:0x101EBC10, sys_thread_t:0x41FB6D60, state:CW, native ID:0x15004A) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at com.javtech.utils.LiteBlockingQueue.get(LiteBlockingQueue.java:54)
            4XESTACKTRACE at com.javtech.tradescope.alert.b.run(TradeScopeUI:75)
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 41FB704C, EBX : 41FB6D60, ECX : 403DAE00
            3HPREGVALUES EDX : 41FB704C, ESI : FFFFFFFF, EDI : 08C2C0D8
            3HPREGVALUES EBP : B67FF6E4, ESP : B67FF6AC, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-63" PID 15693
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-61" (TID:0x101CDFC8, sys_thread_t:0x41F14BD8, state:CW, native ID:0x148048) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.util.TimerThread.mainLoop(Timer.java(Compiled Code))
            4XESTACKTRACE at java.util.TimerThread.run(Timer.java:396)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 41F14EC4, EBX : 41F14BD8, ECX : 403DAE00
            3HPREGVALUES EDX : 41F14EC4, ESI : 00000000, EDI : 08C2BAD8
            3HPREGVALUES EBP : B6BFF708, ESP : B6BFF6D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-61" PID 15691
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE 0
            NULL
            3XMTHREADINFO "RMI RenewClean-[192.168.140.22:1098]" (TID:0x101C1A88, sys_thread_t:0x41F2AA30, state:CW, native ID:0x138013) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java(Compiled Code))
            4XESTACKTRACE at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:517)
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 41F2AD1C, EBX : 41F2AA30, ECX : 403DAE00
            3HPREGVALUES EDX : 41F2AD1C, ESI : 7FFFFF04, EDI : 08C98540
            3HPREGVALUES EBP : BD5FF6D0, ESP : BD5FF698, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "RMI RenewClean-[192.168.140.22:1098]" PID 15687
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-2" (TID:0x101BCC98, sys_thread_t:0x86F23A0, state:CW, native ID:0x130012) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 086F268C, EBX : 086F23A0, ECX : 403DAE00
            3HPREGVALUES EDX : 086F268C, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : BD7FF720, ESP : BD7FF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-2" PID 15685
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor8" (TID:0x1016E3F8, sys_thread_t:0x8CF1F48, state:CW, native ID:0x128046) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08CF2234, EBX : 08CF1F48, ECX : 403DAE00
            3HPREGVALUES EDX : 08CF2234, ESI : FFFFFFFF, EDI : 08C98480
            3HPREGVALUES EBP : B6FFF608, ESP : B6FFF5D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor8" PID 15682
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "TP-Processor7" (TID:0x1016E468, sys_thread_t:0x8CF09E8, state:R, native ID:0x124045) prio=5
            4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
            4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java(Compiled Code))
            4XESTACKTRACE at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : FFFFFFFC, EBX : B71FF428, ECX : 00000001
            3HPREGVALUES EDX : 00004E20, ESI : 0830DC3C, EDI : B71FF428
            3HPREGVALUES EBP : B71FF3B0, ESP : B71FF3A4, EIP : 401392C0
            3HPREGVALUES EFLAGS : 00000212
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor7" PID 15681
            NULL -------------------------
            3HPSTACKLINE __poll at 401392C0 in libc.so.6
            3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
            3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so
            3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
            3HPSTACKLINE 40F415BF
            NULL
            3XMTHREADINFO "TP-Processor6" (TID:0x1015CAC0, sys_thread_t:0x8CEF488, state:R, native ID:0x120044) prio=5
            4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
            4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Compiled Code))
            4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java(Compiled Code))
            4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : B73FF4E8
            3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 08CEF2E8
            3HPREGVALUES EBP : B73FF510, ESP : B73FF4E4, EIP : 40140D80
            3HPREGVALUES EFLAGS : 00000292
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor6" PID 15680
            NULL -------------------------
            3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
            3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
            3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
            3HPSTACKLINE 40F15152
            NULL
            3XMTHREADINFO "TP-Processor5" (TID:0x1015CB30, sys_thread_t:0x8CE9880, state:CW, native ID:0x11C043) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08CE9B6C, EBX : 08CE9880, ECX : 403DAE00
            3HPREGVALUES EDX : 08CE9B6C, ESI : FFFFFFFF, EDI : 08C2BA78
            3HPREGVALUES EBP : B75FF608, ESP : B75FF5D0, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "TP-Processor5" PID 15679
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Message Pushers-1" (TID:0x100B9170, sys_thread_t:0x8C39708, state:CW, native ID:0xF0038) prio=5
            4XESTACKTRACE at java.lang.Object.wait(Native Method)
            4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 08C399F4, EBX : 08C39708, ECX : 403DAE00
            3HPREGVALUES EDX : 08C399F4, ESI : FFFFFFFF, EDI : 08C2BA18
            3HPREGVALUES EBP : B8BFF720, ESP : B8BFF6E8, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Message Pushers-1" PID 15667
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE 0
            3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
            NULL
            3XMTHREADINFO "Thread-24" (TID:0x1009EAF8, sys_thread_t:0x876C7E8, state:R, native ID:0xB4029) prio=5
            4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
            4XESTACKTRACE at com.javtech.tradescope.monitor.MessageCacheMonitor.run(MessageCacheMonitor.java(Compiled Code))
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 0876CAD4, EBX : 0876C7E8, ECX : 00000001
            3HPREGVALUES EDX : 0876CAD4, ESI : 00002710, EDI : 0805CFE8
            3HPREGVALUES EBP : BA9FF744, ESP : BA9FF70C, EIP : 403CDC09
            3HPREGVALUES EFLAGS : 00000246
            NULL
            3HPNATIVESTACK Native Stack of "Thread-24" PID 15652
            NULL -------------------------
            3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
            3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
            NULL
            3XMTHREADINFO "UILServerILService Accept Thread" (TID:0x100A0F28, sys_thread_t:0x872CDC0, state:R, native ID:0xB0028) prio=5
            4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
            4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Compiled Code))
            4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java(Compiled Code))
            4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java(Compiled Code))
            4XESTACKTRACE at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : BABFF6D8
            3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 0872CC20
            3HPREGVALUES EBP : BABFF700, ESP : BABFF6D4, EIP : 40140D80
            3HPREGVALUES EFLAGS : 00000296
            NULL
            3HPNATIVESTACK Native Stack of "UILServerILService Accept Thread" PID 15651
            NULL -------------------------
            3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
            3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
            3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
            3HPSTACKLINE 40F15152
            NULL
            3XMTHREADINFO "RMI TCP Accept-0" (TID:0x100605B0, sys_thread_t:0x8C1B188, state:R, native ID:0xAC027) prio=5
            4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
            4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:374)
            4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java:460)
            4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java:431)
            4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:351)
            4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
            NULL
            3HPREGISTERS Register Values
            NULL ---------------
            3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : BADFF674
            3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 08C1AFE8
            3HPREGVALUES EBP : BADFF69C, ESP : BADFF670, EIP : 40140D80
            3HPREGVALUES EFLAGS : 00000292
            NULL
            3HPNATIVESTACK Native Stack of "RMI TCP Accept-0" PID 15650
            NULL -------------------------
            3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
            3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
            3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
            3HPSTACKLINE 40F15152
            3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF03 in libjvm.so
            3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF03 in libjvm.so
            3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF0E in libjvm.so
            3HPSTACKLINE L0_invokeinterface_quick__ at 4031F39A in libjvm.so
            3HPSTACKLINE mmipExecuteJava at 40319581 in libjvm.so
            3HPSTACKLINE BADFF984
            NULL
            3XMTHREADINFO "OIL Worker Server" (TID:0x100623F0, sys_thread_t:0x8C0E830, state:R, native ID:0xA4025) prio=5
            4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
            4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Co

            • 3. Re: JMS hang/crash (Jboss 3.2.3)
              Adrian Brock Master

              The VM state will always be unpredictable after an OutOfMemoryError.
              You need to figure out why you got this error. What was using all the memory?

              Regards,
              Adrian

              • 4. Re: JMS hang/crash (Jboss 3.2.3)
                Richard Bielak Newbie

                 

                "adrian@jboss.org" wrote:
                The VM state will always be unpredictable after an OutOfMemoryError.
                You need to figure out why you got this error. What was using all the memory?

                Regards,
                Adrian


                After the error, the web-console was working fine and it showed that there was plenty of memory. The web part of our app was working.

                Some of our internal threads had died and JMS was hosed...

                ...richie

                • 5. Re: JMS hang/crash (Jboss 3.2.3)
                  Richard Bielak Newbie

                  Re: OutOfMemoryError. This is most likely due to our app.

                  However, we saw some indication that JMS wasn't healthy little before the crash.

                  We have a monitor that watches msg counts on our topics and one of the topics was showing some pending messages (more than 100). The normal count should be 0.

                  ...richie

                  • 6. Re: JMS hang/crash (Jboss 3.2.3)
                    Adrian Brock Master

                    If memory is tight, JBossMQ will push messages onto disk rather than holding them
                    in memory. If memory is tight for a long time, you will find JBossMQ does not perform
                    very well. It will be running virtually entirely from disk.

                    Have a look at the SoftenedSize on the MessageCache to see how messages
                    JBossMQ is pushing onto disk. A large number indicates you have the problem
                    described above.

                    You should also make sure the high and max memorymark are consistent
                    with your available memory.

                    Regards,
                    Adrian

                    • 7. Re: JMS hang/crash (Jboss 3.2.3)
                      Michael Kerry Newbie

                       

                      "richieb" wrote:


                      We have a monitor that watches msg counts on our topics and one of the topics was showing some pending messages (more than 100). The normal count should be 0.

                      ...richie


                      Slightly off topic (pardon the pun) but I would be interested in seeing how you are monitoring the topic count programmatically. Could you post some code?

                      Thanks,
                      -MK

                      • 8. Re: JMS hang/crash (Jboss 3.2.3)
                        Mike Hemling Newbie

                        I've had the EXACT same problem running java 1.4.2 on Red Hat 9. We have a ton of memory. I am still not sure what caused it but using the LD_ASSUME_KERNEL=2.4.1 that Adrian has talked about appears to have fixed my problems. Still not sure how but you may want to at least look into it.

                        Mike

                        • 9. Re: JMS hang/crash (Jboss 3.2.3)
                          Richard Bielak Newbie

                          Just to put this topic to rest, we discovered a deadlock in our application code between an onMessage routine and some other code.

                          Basically, when a routine got a JMS exception it would try to reconnect to JMS, which would hang until all callbacks were done. A a callback did a JMS call that got an exception and tried to restart connection, but would wait until the restart in progress was done. :)

                          Thanks for all your suggestions.

                          ...richie

                          • 10. Re: JMS hang/crash (Jboss 3.2.3)
                            Richard Bielak Newbie

                             

                            "mkerry" wrote:


                            Slightly off topic (pardon the pun) but I would be interested in seeing how you are monitoring the topic count programmatically. Could you post some code?

                            Thanks,
                            -MK


                            We wrote a simple JMX service that would use JMX services to check the topics. Here is a snippet of the code:

                            package monitor;

                            import org.jboss.jmx.adaptor.rmi.RMIAdaptor;
                            import org.apache.log4j.Logger;
                            import javax.naming.InitialContext;
                            import javax.management.ObjectName;

                            public class MessageCacheMonitor implements MessageCacheMonitorMBean, Runnable {

                            private int maxMessageThreshold_ = 0;
                            private int sleepInterval_ = 10000;
                            private RMIAdaptor server_;
                            private ObjectName heartbeatTopic_;
                            private Thread runner_ = null;
                            private Logger logger_ = MessageCacheMonitorLogger.getMainLogger();
                            private boolean done_ = false;

                            public int getMaxMessageThreshold()
                            {
                            return maxMessageThreshold_;
                            }

                            public void setMaxMessageThreshold(int maxMessageThreshold)
                            {
                            maxMessageThreshold_ = maxMessageThreshold;
                            logger_.info("Setting Message Cache Monitor max message threshold to: " + maxMessageThreshold);
                            }

                            public int getSleepInterval() {
                            return sleepInterval_;
                            }

                            public void setSleepInterval(int millis) {
                            sleepInterval_ = millis;
                            logger_.info("Setting Message Cache Monitor sleep interval to: " + millis);
                            }

                            public String getLogLevel() {
                            return MessageCacheMonitorLogger.getLogLevel();
                            }

                            public void setLogLevel(String logLevel) {
                            MessageCacheMonitorLogger.setLogLevel(logLevel);
                            logger_ = MessageCacheMonitorLogger.getMainLogger();
                            logger_.info("Set Message Cache Monitor log level to: " + getLogLevel());
                            }

                            public void run() {
                            while (!done_) {
                            try {
                            int messageCount = ((Integer)server_.getAttribute(heartbeatTopic_, "AllMessageCount")).intValue();
                            logger_.info("Message Cache Monitor: Message Count for Heartbeat Topic is " + messageCount);
                            logger_.debug("Message Cache Monitor max message threshold is " + maxMessageThreshold_);

                            if (messageCount >= maxMessageThreshold_) {

                            logger_.warn("Message count for tradescope heartbeat topic has exceeded configured threshold of " +

                            maxMessageThreshold_ + ". Purging all messages.");

                            }

                            } catch (Exception e) {
                            logger_.error("Exception in MessageCacheMonitor", e);

                            }

                            try {
                            Thread.sleep(sleepInterval_);
                            } catch (Exception e) {
                            }

                            }

                            }



                            public void create() throws Exception {
                            InitialContext ic = new InitialContext();
                            server_ = (RMIAdaptor) ic.lookup("jmx/rmi/RMIAdaptor");
                            logger_.info("MessageCacheMonitor found RMIAdaptor: " + server_);
                            heartbeatTopic_ = new ObjectName("tradescope.destination:name=ts.agent.heartbeat,service=Topic");
                            logger_.info("Found heartbeat topic " + heartbeatTopic_);
                            }



                            public void start() throws Exception {
                            if (runner_ != null)
                            return;
                            done_ = false;
                            runner_ = new Thread(this);
                            runner_.start();
                            }

                            public void stop() {
                            if (runner_ == null)
                            return;
                            done_ = true;
                            runner_.interrupt();
                            runner_ = null;

                            }

                            public void destroy() {
                            stop();
                            }

                            }