10 Replies Latest reply on Mar 29, 2004 1:11 PM by richieb

JMS hang/crash (Jboss 3.2.3)

richieb Mar 17, 2004 4:40 PM

We had a JMS crash/hang occur in our development JBoss 3.2.3. An OutOfMemory error occured first, and then the clients got into a weird state. Some clients got stuck while trying to reconnect to Jboss in InitialContext look up.

We had a program that was monitoring topic message counts and it started to report problems few minutes before.

Here is first incation of problem in the log:

2004-03-17 10:29:01,635 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.AddMsg791646028[msgTy
pe: m_addMessage, msgID: 430942, error: null]
java.lang.OutOfMemoryError
2004-03-17 10:29:01,635 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.uil2.msgs.AddMsg274059084[msgTy
pe: m_addMessage, msgID: 1163451, error: null]
java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java(Compiled Code))
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java(Inlined Compiled Code))
at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java(Compiled Code))
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java(Compiled Code))
at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java(Compiled Code))
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java(Compiled Code))
at java.lang.Thread.run(Thread.java:568)
2004-03-17 10:45:19,321 WARN [com.javtech.tradescope] JMS service not reachable ... will try again in a few seconds
2004-03-17 10:42:25,503 ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] Endpoint ServerSocket[addr=/0.0.0.0,port=0,localport=8081
] ignored exception: java.net.SocketException: Connection reset by peer

And then we see lots of these:

2004-03-17 11:10:34,387 ERROR [STDERR] java.lang.OutOfMemoryError:
2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:68)
2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenera
tor.java:111)
2004-03-17 11:10:34,387 ERROR [STDERR] at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:317)
2004-03-17 11:10:34,387 ERROR [STDERR] at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java(Compiled Code
))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.access$1100(ObjectStreamClass.java:72)
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:399)
2004-03-17 11:10:34,388 ERROR [STDERR] at java.security.AccessController.doPrivileged(Native Method)
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.(ObjectStreamClass.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java(Inlined Compiled Co
de))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java(Compiled Code))
2004-03-17 11:10:34,388 ERROR [STDERR] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java(Compiled Code))
2004-03-17 11:10:34,389 ERROR [STDERR] at org.jnp.server.Main.run(Main.java:306)
2004-03-17 11:10:34,389 ERROR [STDERR] at java.lang.Thread.run(Thread.java(Compiled Code))
2004-03-17 11:10:34,389 ERROR [STDERR] java.lang.OutOfMemoryError:

I took thread dump while the server was in this state. It was from IBM's JVM running on Linux. Please let me know where to post it.

...richie

1. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 17, 2004 4:44 PM (in response to richieb)

I forgot to add that when I tried to shut Jboss down with shutdown.sh
script it did not work. The script timed out.

..richie
Actions
2. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 18, 2004 7:22 AM (in response to richieb)

Here is the extract ot the threaddump at the time JMS was stuck:

2XMFULLTHDDUMP Full thread dump Classic VM (J2RE 1.4.1 IBM build cxia32141-20030522, native threads):
3XMTHREADINFO "JNP Server" (TID:0x1051DE78, sys_thread_t:0x8C2E2F0, state:R, native ID:0xBCA4000F) prio=5
4XESTACKTRACE at java.lang.Thread.start(Native Method)
4XESTACKTRACE at org.jnp.server.Main.listen(Main.java(Compiled Code))
4XESTACKTRACE at org.jnp.server.Main.run(Main.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 00000000, EBX : 08C2E898, ECX : 08C2E8BC
3HPREGVALUES EDX : 00000000, ESI : 08C2E8BC, EDI : 00000000
3HPREGVALUES EBP : BDDFF70C, ESP : BDDFF6C0, EIP : 4012D9F7
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "JNP Server" PID 21907
NULL -------------------------
3HPSTACKLINE __sched_yield at 4012D9F7 in libc.so.6
3HPSTACKLINE xeThreadCreate at 4034F2B1 in libjvm.so
3HPSTACKLINE xmCreateThread at 40353E06 in libjvm.so
3HPSTACKLINE JVM_StartThread at 4025FBD4 in libjvm.so
3HPSTACKLINE 40BD608E
3HPSTACKLINE 0
NULL
3XMTHREADINFO "MessageListenerThread - ts.toserver.engineinfo" (TID:0x1050F300, sys_thread_t:0x8F45178, state:CW, native ID:0xBC95C022) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08F45464, EBX : 08F45178, ECX : 403DAE00
3HPREGVALUES EDX : 08F45464, ESI : FFFFFFFF, EDI : 08ED6710
3HPREGVALUES EBP : BB7FF750, ESP : BB7FF718, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.toserver.engineinfo" PID 21850
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "MessageListenerThread - ts.toserver.masterconfig" (TID:0x1050F410, sys_thread_t:0x8F51FD0, state:CW, native ID:0xBC954020) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08F522BC, EBX : 08F51FD0, ECX : 403DAE00
3HPREGVALUES EDX : 08F522BC, ESI : FFFFFFFF, EDI : 08EDF530
3HPREGVALUES EBP : BBBFF750, ESP : BBBFF718, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.toserver.masterconfig" PID 21848
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "MessageListenerThread - ts.agent.heartbeat" (TID:0x1050EA50, sys_thread_t:0x8F44BD0, state:CW, native ID:0xBC94C01F) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08F44EBC, EBX : 08F44BD0, ECX : 403DAE00
3HPREGVALUES EDX : 08F44EBC, ESI : FFFFFFFF, EDI : 08EE5A10
3HPREGVALUES EBP : BBDFF750, ESP : BBDFF718, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.agent.heartbeat" PID 21846
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "MessageListenerThread - ts.fromserver.alert" (TID:0x1050EB60, sys_thread_t:0x875B888, state:CW, native ID:0xBC944007) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 0875BB74, EBX : 0875B888, ECX : 403DAE00
3HPREGVALUES EDX : 0875BB74, ESI : FFFFFFFF, EDI : 08EE3A90
3HPREGVALUES EBP : BEDFF750, ESP : BEDFF718, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.fromserver.alert" PID 21844
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "MessageListenerThread - ts.agent.command" (TID:0x1050EC70, sys_thread_t:0x92B1188, state:CW, native ID:0xBC93C005) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.SpyMessageConsumer.run(SpyMessageConsumer.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 092B1474, EBX : 092B1188, ECX : 403DAE00
3HPREGVALUES EDX : 092B1474, ESI : FFFFFFFF, EDI : 08EBAA48
3HPREGVALUES EBP : BF1FF750, ESP : BF1FF718, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "MessageListenerThread - ts.agent.command" PID 21842
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "RMI ConnectionExpiration-[192.168.140.22:1098]" (TID:0x102B4008, sys_thread_t:0x9186A90, state:R, native ID:0x7758C014) prio=5
4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
4XESTACKTRACE at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 09186D7C, EBX : 09186A90, ECX : 00000001
3HPREGVALUES EDX : 09186D7C, ESI : 00003A98, EDI : 0805CFE8
3HPREGVALUES EBP : BD3FF78C, ESP : BD3FF754, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "RMI ConnectionExpiration-[192.168.140.22:1098]" PID 12464
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
NULL
3XMTHREADINFO "RMI TCP Connection(21636)-192.168.140.22" (TID:0x102B40B8, sys_thread_t:0x42C01988, state:R, native ID:0x7758400E) prio=5
4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
4XESTACKTRACE at java.io.BufferedInputStream.fill(BufferedInputStream.java(Compiled Code))
4XESTACKTRACE at java.io.BufferedInputStream.read(BufferedInputStream.java(Compiled Code))
4XESTACKTRACE at java.io.FilterInputStream.read(FilterInputStream.java(Compiled Code))
4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java(Compiled Code))
4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : FFFFFFFC, EBX : BDFFF4C8, ECX : 00000001
3HPREGVALUES EDX : 006DDD00, ESI : 0830DEC0, EDI : BDFFF4C8
3HPREGVALUES EBP : BDFFF450, ESP : BDFFF444, EIP : 401392C0
3HPREGVALUES EFLAGS : 00000212
NULL
3HPNATIVESTACK Native Stack of "RMI TCP Connection(21636)-192.168.140.22" PID 12462
NULL -------------------------
3HPSTACKLINE __poll at 401392C0 in libc.so.6
3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so

3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
3HPSTACKLINE 40F415BF
NULL
3XMTHREADINFO "Thread-64290" (TID:0x113C2F08, sys_thread_t:0x43316370, state:CW, native ID:0x536F80C2) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4331665C, EBX : 43316370, ECX : 43316370
3HPREGVALUES EDX : 4331665C, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A77FF648, ESP : A77FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64290" PID 6127
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64289" (TID:0x113C3010, sys_thread_t:0x43315DC8, state:CW, native ID:0x536F00C0) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 433160B4, EBX : 43315DC8, ECX : 43315DC8
3HPREGVALUES EDX : 433160B4, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A7BFF648, ESP : A7BFF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64289" PID 6121
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64288" (TID:0x113C3068, sys_thread_t:0x43315820, state:R, native ID:0x536EC0BF) prio=5
4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:193)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 43315B0C, EBX : 43315820, ECX : 00000001
3HPREGVALUES EDX : 43315B0C, ESI : 00002710, EDI : 0805CFE8
3HPREGVALUES EBP : A7DFF5E0, ESP : A7DFF5A8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64288" PID 6120
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
NULL
3XMTHREADINFO "Thread-64287" (TID:0x113C31C8, sys_thread_t:0x43313308, state:CW, native ID:0x536E00BC) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 433135F4, EBX : 43313308, ECX : 43313308
3HPREGVALUES EDX : 433135F4, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A83FF648, ESP : A83FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64287" PID 6117
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64286" (TID:0x113C32D0, sys_thread_t:0x43312D60, state:CW, native ID:0x536D40BA) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4331304C, EBX : 43312D60, ECX : 43312D60
3HPREGVALUES EDX : 4331304C, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A87FF648, ESP : A87FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64286" PID 6114
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64285" (TID:0x113C3328, sys_thread_t:0x43307168, state:CW, native ID:0x536CC0A6) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 43307454, EBX : 43307168, ECX : 43307168
3HPREGVALUES EDX : 43307454, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : AAFFF648, ESP : AAFFF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64285" PID 5884
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64284" (TID:0x113C3430, sys_thread_t:0x433127B8, state:CW, native ID:0x536C00B8) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 43312AA4, EBX : 433127B8, ECX : 433127B8
3HPREGVALUES EDX : 43312AA4, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A8BFF648, ESP : A8BFF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64284" PID 5881
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64280" (TID:0x106983E8, sys_thread_t:0x43312210, state:CW, native ID:0x536A80B6) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 433124FC, EBX : 43312210, ECX : 43312210
3HPREGVALUES EDX : 433124FC, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A8FFF648, ESP : A8FFF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64280" PID 5871
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64279" (TID:0x106984F0, sys_thread_t:0x43307CB8, state:CW, native ID:0x5369C0B4) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 43307FA4, EBX : 43307CB8, ECX : 43307CB8
3HPREGVALUES EDX : 43307FA4, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A93FF648, ESP : A93FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64279" PID 5865
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64278" (TID:0x106985F8, sys_thread_t:0x4330B730, state:CW, native ID:0x536900B1) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4330BA1C, EBX : 4330B730, ECX : 4330B730
3HPREGVALUES EDX : 4330BA1C, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : A99FF648, ESP : A99FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64278" PID 5862
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64277" (TID:0x10698700, sys_thread_t:0x4330DC48, state:CW, native ID:0x53684071) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4330DF34, EBX : 4330DC48, ECX : 4330DC48
3HPREGVALUES EDX : 4330DF34, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : B19FF648, ESP : B19FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64277" PID 5859
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64274" (TID:0x10293CB8, sys_thread_t:0x4330E1F0, state:CW, native ID:0x536600A0) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4330E4DC, EBX : 4330E1F0, ECX : 4330E1F0
3HPREGVALUES EDX : 4330E4DC, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : ABBFF648, ESP : ABBFF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64274" PID 5850
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-64272" (TID:0x1066E7D8, sys_thread_t:0x4330C6E8, state:CW, native ID:0x53654082) prio=5
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI:180)
4XESTACKTRACE at com.javtech.tradescope.jms.g.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller.a(TradeScopeUI(Compiled Code))
4XESTACKTRACE at com.javtech.tradescope.model.ActiveIntervalEnginePoller$1.run(TradeScopeUI(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4330C9D4, EBX : 4330C6E8, ECX : 4330C6E8
3HPREGVALUES EDX : 4330C9D4, ESI : 101BCB68, EDI : 08EE4B10
3HPREGVALUES EBP : AF7FF648, ESP : AF7FF610, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-64272" PID 5847
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor16" (TID:0x1032B228, sys_thread_t:0x8FE93B0, state:CW, native ID:0x7B6008A) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08FE969C, EBX : 08FE93B0, ECX : 403DAE00
3HPREGVALUES EDX : 08FE969C, ESI : FFFFFFFF, EDI : 08D37070
3HPREGVALUES EBP : AE7FF630, ESP : AE7FF5F8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor16" PID 2858
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor15" (TID:0x1032B298, sys_thread_t:0x8FE8E08, state:CW, native ID:0x7B5C089) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08FE90F4, EBX : 08FE8E08, ECX : 403DAE00
3HPREGVALUES EDX : 08FE90F4, ESI : FFFFFFFF, EDI : 08E8B190
3HPREGVALUES EBP : AE9FF630, ESP : AE9FF5F8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor15" PID 2857
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor14" (TID:0x1032B308, sys_thread_t:0x8FE52E0, state:CW, native ID:0x7B58088) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08FE55CC, EBX : 08FE52E0, ECX : 403DAE00
3HPREGVALUES EDX : 08FE55CC, ESI : FFFFFFFF, EDI : 08EA0440
3HPREGVALUES EBP : AEBFF630, ESP : AEBFF5F8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor14" PID 2856
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor13" (TID:0x1032B378, sys_thread_t:0x8FE4D38, state:CW, native ID:0x7B54086) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08FE5024, EBX : 08FE4D38, ECX : 403DAE00
3HPREGVALUES EDX : 08FE5024, ESI : FFFFFFFF, EDI : 08EDBC30
3HPREGVALUES EBP : AEFFF630, ESP : AEFFF5F8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor13" PID 2855
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-10" (TID:0x10207A70, sys_thread_t:0x8DC0218, state:CW, native ID:0x300073) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08DC0504, EBX : 08DC0218, ECX : 403DAE00
3HPREGVALUES EDX : 08DC0504, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B15FF720, ESP : B15FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-10" PID 15867
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-9" (TID:0x101F9C88, sys_thread_t:0x8CFB890, state:CW, native ID:0x2C4056) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08CFBB7C, EBX : 08CFB890, ECX : 403DAE00
3HPREGVALUES EDX : 08CFBB7C, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B4FFF720, ESP : B4FFF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-9" PID 15820
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-8" (TID:0x101A3340, sys_thread_t:0x8E848C0, state:CW, native ID:0x2B0053) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08E84BAC, EBX : 08E848C0, ECX : 403DAE00
3HPREGVALUES EDX : 08E84BAC, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B55FF720, ESP : B55FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-8" PID 15812
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-7" (TID:0x101FBD00, sys_thread_t:0x8E715E0, state:CW, native ID:0x28C03B) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08E718CC, EBX : 08E715E0, ECX : 403DAE00
3HPREGVALUES EDX : 08E718CC, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B85FF720, ESP : B85FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-7" PID 15776
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-6" (TID:0x10106C18, sys_thread_t:0x8C9EE48, state:CW, native ID:0x25C035) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08C9F134, EBX : 08C9EE48, ECX : 403DAE00
3HPREGVALUES EDX : 08C9F134, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B91FF720, ESP : B91FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-6" PID 15764
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-131" (TID:0x100603D0, sys_thread_t:0x42025320, state:CW, native ID:0x24C023) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java:438)
4XESTACKTRACE at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 4202560C, EBX : 42025320, ECX : 403DAE00
3HPREGVALUES EDX : 4202560C, ESI : FFFFFFFF, EDI : 0805B9C0
3HPREGVALUES EBP : BB5FF720, ESP : BB5FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-131" PID 15760
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-5" (TID:0x101B3948, sys_thread_t:0x8DE0970, state:CW, native ID:0x23006D) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08DE0C5C, EBX : 08DE0970, ECX : 403DAE00
3HPREGVALUES EDX : 08DE0C5C, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B21FF720, ESP : B21FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-5" PID 15752
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor12" (TID:0x10222B28, sys_thread_t:0x8DA1A90, state:R, native ID:0x1AC02F) prio=5
4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : FFFFFFFC, EBX : B9DFF428, ECX : 00000001
3HPREGVALUES EDX : 00004E20, ESI : 0830DC20, EDI : B9DFF428
3HPREGVALUES EBP : B9DFF3B0, ESP : B9DFF3A4, EIP : 401392C0
3HPREGVALUES EFLAGS : 00000212
NULL
3HPNATIVESTACK Native Stack of "TP-Processor12" PID 15717
NULL -------------------------
3HPSTACKLINE __poll at 401392C0 in libc.so.6
3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so
3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
3HPSTACKLINE 40F415BF
NULL
3XMTHREADINFO "TP-Processor11" (TID:0x10222C48, sys_thread_t:0x8713378, state:CW, native ID:0x1A002A) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08713664, EBX : 08713378, ECX : 403DAE00
3HPREGVALUES EDX : 08713664, ESI : FFFFFFFF, EDI : 08C985A0
3HPREGVALUES EBP : BA7FF608, ESP : BA7FF5D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor11" PID 15714
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor10" (TID:0x10222D10, sys_thread_t:0x8C5A478, state:CW, native ID:0x19803A) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08C5A764, EBX : 08C5A478, ECX : 403DAE00
3HPREGVALUES EDX : 08C5A764, ESI : FFFFFFFF, EDI : 08D36A70
3HPREGVALUES EBP : B87FF608, ESP : B87FF5D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor10" PID 15712
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor9" (TID:0x10222DD8, sys_thread_t:0x8D7B960, state:CW, native ID:0x19000D) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08D7BC4C, EBX : 08D7B960, ECX : 403DAE00
3HPREGVALUES EDX : 08D7BC4C, ESI : FFFFFFFF, EDI : 08D36A10
3HPREGVALUES EBP : BE1FF608, ESP : BE1FF5D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor9" PID 15710
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-4" (TID:0x101FAB70, sys_thread_t:0x8CFD4E0, state:CW, native ID:0x17C055) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08CFD7CC, EBX : 08CFD4E0, ECX : 403DAE00
3HPREGVALUES EDX : 08CFD7CC, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B51FF720, ESP : B51FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-4" PID 15704
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-3" (TID:0x101FAC40, sys_thread_t:0x8CFBF80, state:CW, native ID:0x178054) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08CFC26C, EBX : 08CFBF80, ECX : 403DAE00
3HPREGVALUES EDX : 08CFC26C, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B53FF720, ESP : B53FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-3" PID 15703
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-63" (TID:0x101EBC10, sys_thread_t:0x41FB6D60, state:CW, native ID:0x15004A) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at com.javtech.utils.LiteBlockingQueue.get(LiteBlockingQueue.java:54)
4XESTACKTRACE at com.javtech.tradescope.alert.b.run(TradeScopeUI:75)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 41FB704C, EBX : 41FB6D60, ECX : 403DAE00
3HPREGVALUES EDX : 41FB704C, ESI : FFFFFFFF, EDI : 08C2C0D8
3HPREGVALUES EBP : B67FF6E4, ESP : B67FF6AC, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-63" PID 15693
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-61" (TID:0x101CDFC8, sys_thread_t:0x41F14BD8, state:CW, native ID:0x148048) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.util.TimerThread.mainLoop(Timer.java(Compiled Code))
4XESTACKTRACE at java.util.TimerThread.run(Timer.java:396)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 41F14EC4, EBX : 41F14BD8, ECX : 403DAE00
3HPREGVALUES EDX : 41F14EC4, ESI : 00000000, EDI : 08C2BAD8
3HPREGVALUES EBP : B6BFF708, ESP : B6BFF6D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-61" PID 15691
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE 0
NULL
3XMTHREADINFO "RMI RenewClean-[192.168.140.22:1098]" (TID:0x101C1A88, sys_thread_t:0x41F2AA30, state:CW, native ID:0x138013) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java(Compiled Code))
4XESTACKTRACE at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:517)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 41F2AD1C, EBX : 41F2AA30, ECX : 403DAE00
3HPREGVALUES EDX : 41F2AD1C, ESI : 7FFFFF04, EDI : 08C98540
3HPREGVALUES EBP : BD5FF6D0, ESP : BD5FF698, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "RMI RenewClean-[192.168.140.22:1098]" PID 15687
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-2" (TID:0x101BCC98, sys_thread_t:0x86F23A0, state:CW, native ID:0x130012) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 086F268C, EBX : 086F23A0, ECX : 403DAE00
3HPREGVALUES EDX : 086F268C, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : BD7FF720, ESP : BD7FF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-2" PID 15685
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor8" (TID:0x1016E3F8, sys_thread_t:0x8CF1F48, state:CW, native ID:0x128046) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08CF2234, EBX : 08CF1F48, ECX : 403DAE00
3HPREGVALUES EDX : 08CF2234, ESI : FFFFFFFF, EDI : 08C98480
3HPREGVALUES EBP : B6FFF608, ESP : B6FFF5D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor8" PID 15682
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "TP-Processor7" (TID:0x1016E468, sys_thread_t:0x8CF09E8, state:R, native ID:0x124045) prio=5
4XESTACKTRACE at java.net.SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java.net.SocketInputStream.read(SocketInputStream.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java(Compiled Code))
4XESTACKTRACE at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : FFFFFFFC, EBX : B71FF428, ECX : 00000001
3HPREGVALUES EDX : 00004E20, ESI : 0830DC3C, EDI : B71FF428
3HPREGVALUES EBP : B71FF3B0, ESP : B71FF3A4, EIP : 401392C0
3HPREGVALUES EFLAGS : 00000212
NULL
3HPNATIVESTACK Native Stack of "TP-Processor7" PID 15681
NULL -------------------------
3HPSTACKLINE __poll at 401392C0 in libc.so.6
3HPSTACKLINE NET_Poll at 40D8EFD3 in libnet.so
3HPSTACKLINE NET_Timeout at 40D8EF0B in libnet.so
3HPSTACKLINE Java_java_net_SocketInputStream_socketRead0 at 40D8DDA9 in libnet.so
3HPSTACKLINE 40F415BF
NULL
3XMTHREADINFO "TP-Processor6" (TID:0x1015CAC0, sys_thread_t:0x8CEF488, state:R, native ID:0x120044) prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : B73FF4E8
3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 08CEF2E8
3HPREGVALUES EBP : B73FF510, ESP : B73FF4E4, EIP : 40140D80
3HPREGVALUES EFLAGS : 00000292
NULL
3HPNATIVESTACK Native Stack of "TP-Processor6" PID 15680
NULL -------------------------
3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
3HPSTACKLINE 40F15152
NULL
3XMTHREADINFO "TP-Processor5" (TID:0x1015CB30, sys_thread_t:0x8CE9880, state:CW, native ID:0x11C043) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08CE9B6C, EBX : 08CE9880, ECX : 403DAE00
3HPREGVALUES EDX : 08CE9B6C, ESI : FFFFFFFF, EDI : 08C2BA78
3HPREGVALUES EBP : B75FF608, ESP : B75FF5D0, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "TP-Processor5" PID 15679
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Message Pushers-1" (TID:0x100B9170, sys_thread_t:0x8C39708, state:CW, native ID:0xF0038) prio=5
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.idle(ThreadPool.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.threadpool.ThreadPool$WorkerThread.run(ThreadPool.java(Compiled Code))
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 08C399F4, EBX : 08C39708, ECX : 403DAE00
3HPREGVALUES EDX : 08C399F4, ESI : FFFFFFFF, EDI : 08C2BA18
3HPREGVALUES EBP : B8BFF720, ESP : B8BFF6E8, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Message Pushers-1" PID 15667
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE 0
3HPSTACKLINE _hpiEnterProtectedSection at 403CB9EF in libhpi.so
NULL
3XMTHREADINFO "Thread-24" (TID:0x1009EAF8, sys_thread_t:0x876C7E8, state:R, native ID:0xB4029) prio=5
4XESTACKTRACE at java.lang.Thread.sleep(Native Method)
4XESTACKTRACE at com.javtech.tradescope.monitor.MessageCacheMonitor.run(MessageCacheMonitor.java(Compiled Code))
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 0876CAD4, EBX : 0876C7E8, ECX : 00000001
3HPREGVALUES EDX : 0876CAD4, ESI : 00002710, EDI : 0805CFE8
3HPREGVALUES EBP : BA9FF744, ESP : BA9FF70C, EIP : 403CDC09
3HPREGVALUES EFLAGS : 00000246
NULL
3HPNATIVESTACK Native Stack of "Thread-24" PID 15652
NULL -------------------------
3HPSTACKLINE _hpiSaveContext at 403CDC09 in libhpi.so
3HPSTACKLINE ?? at 4001CD53 in libpthread.so.0
NULL
3XMTHREADINFO "UILServerILService Accept Thread" (TID:0x100A0F28, sys_thread_t:0x872CDC0, state:R, native ID:0xB0028) prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java(Compiled Code))
4XESTACKTRACE at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : BABFF6D8
3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 0872CC20
3HPREGVALUES EBP : BABFF700, ESP : BABFF6D4, EIP : 40140D80
3HPREGVALUES EFLAGS : 00000296
NULL
3HPNATIVESTACK Native Stack of "UILServerILService Accept Thread" PID 15651
NULL -------------------------
3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
3HPSTACKLINE 40F15152
NULL
3XMTHREADINFO "RMI TCP Accept-0" (TID:0x100605B0, sys_thread_t:0x8C1B188, state:R, native ID:0xAC027) prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:374)
4XESTACKTRACE at java.net.ServerSocket.implAccept(ServerSocket.java:460)
4XESTACKTRACE at java.net.ServerSocket.accept(ServerSocket.java:431)
4XESTACKTRACE at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:351)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
NULL
3HPREGISTERS Register Values
NULL ---------------
3HPREGVALUES EAX : 00000066, EBX : 00000005, ECX : BADFF674
3HPREGVALUES EDX : 40026B50, ESI : 00000000, EDI : 08C1AFE8
3HPREGVALUES EBP : BADFF69C, ESP : BADFF670, EIP : 40140D80
3HPREGVALUES EFLAGS : 00000292
NULL
3HPNATIVESTACK Native Stack of "RMI TCP Accept-0" PID 15650
NULL -------------------------
3HPSTACKLINE __libc_accept at 40140D80 in libc.so.6
3HPSTACKLINE NET_Accept at 40D8EC14 in libnet.so
3HPSTACKLINE Java_java_net_PlainSocketImpl_socketAccept at 40D8CC45 in libnet.so
3HPSTACKLINE 40F15152
3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF03 in libjvm.so
3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF03 in libjvm.so
3HPSTACKLINE L0_invokevirtual_quick__ at 4031EF0E in libjvm.so
3HPSTACKLINE L0_invokeinterface_quick__ at 4031F39A in libjvm.so
3HPSTACKLINE mmipExecuteJava at 40319581 in libjvm.so
3HPSTACKLINE BADFF984
NULL
3XMTHREADINFO "OIL Worker Server" (TID:0x100623F0, sys_thread_t:0x8C0E830, state:R, native ID:0xA4025) prio=5
4XESTACKTRACE at java.net.PlainSocketImpl.socketAccept(Native Method)
4XESTACKTRACE at java.net.PlainSocketImpl.accept(PlainSocketImpl.java(Co
Actions
3. Re: JMS hang/crash (Jboss 3.2.3)

adrian.brock Mar 18, 2004 8:56 AM (in response to richieb)

The VM state will always be unpredictable after an OutOfMemoryError.
You need to figure out why you got this error. What was using all the memory?

Regards,
Adrian
Actions
4. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 18, 2004 9:15 AM (in response to richieb)

"adrian@jboss.org" wrote:
The VM state will always be unpredictable after an OutOfMemoryError.
You need to figure out why you got this error. What was using all the memory?

Regards,
Adrian

After the error, the web-console was working fine and it showed that there was plenty of memory. The web part of our app was working.

Some of our internal threads had died and JMS was hosed...

...richie
Actions
5. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 18, 2004 9:50 AM (in response to richieb)

Re: OutOfMemoryError. This is most likely due to our app.

However, we saw some indication that JMS wasn't healthy little before the crash.

We have a monitor that watches msg counts on our topics and one of the topics was showing some pending messages (more than 100). The normal count should be 0.

...richie
Actions
6. Re: JMS hang/crash (Jboss 3.2.3)

adrian.brock Mar 18, 2004 2:00 PM (in response to richieb)

If memory is tight, JBossMQ will push messages onto disk rather than holding them
in memory. If memory is tight for a long time, you will find JBossMQ does not perform
very well. It will be running virtually entirely from disk.

Have a look at the SoftenedSize on the MessageCache to see how messages
JBossMQ is pushing onto disk. A large number indicates you have the problem
described above.

You should also make sure the high and max memorymark are consistent
with your available memory.

Regards,
Adrian
Actions
7. Re: JMS hang/crash (Jboss 3.2.3)

mkerry Mar 22, 2004 1:52 PM (in response to richieb)

"richieb" wrote:

We have a monitor that watches msg counts on our topics and one of the topics was showing some pending messages (more than 100). The normal count should be 0.

...richie

Slightly off topic (pardon the pun) but I would be interested in seeing how you are monitoring the topic count programmatically. Could you post some code?

Thanks,
-MK
Actions
8. Re: JMS hang/crash (Jboss 3.2.3)

mhemling Mar 23, 2004 9:39 AM (in response to richieb)

I've had the EXACT same problem running java 1.4.2 on Red Hat 9. We have a ton of memory. I am still not sure what caused it but using the LD_ASSUME_KERNEL=2.4.1 that Adrian has talked about appears to have fixed my problems. Still not sure how but you may want to at least look into it.

Mike
Actions
9. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 29, 2004 1:04 PM (in response to richieb)

Just to put this topic to rest, we discovered a deadlock in our application code between an onMessage routine and some other code.

Basically, when a routine got a JMS exception it would try to reconnect to JMS, which would hang until all callbacks were done. A a callback did a JMS call that got an exception and tried to restart connection, but would wait until the restart in progress was done. :)

Thanks for all your suggestions.

...richie
Actions
10. Re: JMS hang/crash (Jboss 3.2.3)

richieb Mar 29, 2004 1:11 PM (in response to richieb)

"mkerry" wrote:

Slightly off topic (pardon the pun) but I would be interested in seeing how you are monitoring the topic count programmatically. Could you post some code?

Thanks,
-MK

We wrote a simple JMX service that would use JMX services to check the topics. Here is a snippet of the code:

package monitor;

import org.jboss.jmx.adaptor.rmi.RMIAdaptor;
import org.apache.log4j.Logger;
import javax.naming.InitialContext;
import javax.management.ObjectName;

public class MessageCacheMonitor implements MessageCacheMonitorMBean, Runnable {

private int maxMessageThreshold_ = 0;
private int sleepInterval_ = 10000;
private RMIAdaptor server_;
private ObjectName heartbeatTopic_;
private Thread runner_ = null;
private Logger logger_ = MessageCacheMonitorLogger.getMainLogger();
private boolean done_ = false;

public int getMaxMessageThreshold()
{
return maxMessageThreshold_;
}

public void setMaxMessageThreshold(int maxMessageThreshold)
{
maxMessageThreshold_ = maxMessageThreshold;
logger_.info("Setting Message Cache Monitor max message threshold to: " + maxMessageThreshold);
}

public int getSleepInterval() {
return sleepInterval_;
}

public void setSleepInterval(int millis) {
sleepInterval_ = millis;
logger_.info("Setting Message Cache Monitor sleep interval to: " + millis);
}

public String getLogLevel() {
return MessageCacheMonitorLogger.getLogLevel();
}

public void setLogLevel(String logLevel) {
MessageCacheMonitorLogger.setLogLevel(logLevel);
logger_ = MessageCacheMonitorLogger.getMainLogger();
logger_.info("Set Message Cache Monitor log level to: " + getLogLevel());
}

public void run() {
while (!done_) {
try {
int messageCount = ((Integer)server_.getAttribute(heartbeatTopic_, "AllMessageCount")).intValue();
logger_.info("Message Cache Monitor: Message Count for Heartbeat Topic is " + messageCount);
logger_.debug("Message Cache Monitor max message threshold is " + maxMessageThreshold_);

if (messageCount >= maxMessageThreshold_) {

logger_.warn("Message count for tradescope heartbeat topic has exceeded configured threshold of " +

maxMessageThreshold_ + ". Purging all messages.");

}

} catch (Exception e) {
logger_.error("Exception in MessageCacheMonitor", e);

}

try {
Thread.sleep(sleepInterval_);
} catch (Exception e) {
}

}

}

public void create() throws Exception {
InitialContext ic = new InitialContext();
server_ = (RMIAdaptor) ic.lookup("jmx/rmi/RMIAdaptor");
logger_.info("MessageCacheMonitor found RMIAdaptor: " + server_);
heartbeatTopic_ = new ObjectName("tradescope.destination:name=ts.agent.heartbeat,service=Topic");
logger_.info("Found heartbeat topic " + heartbeatTopic_);
}

public void start() throws Exception {
if (runner_ != null)
return;
done_ = false;
runner_ = new Thread(this);
runner_.start();
}

public void stop() {
if (runner_ == null)
return;
done_ = true;
runner_.interrupt();
runner_ = null;

}

public void destroy() {
stop();
}

}
Actions

Go to original post