HAJNDI-Listener threads hang
lbasili Sep 26, 2004 5:58 AMHello,
I am using jboss-3.2.3.
My application is running on a partition consisting of two nodes.
I have a client application that uses HAJNDI to balance its requests on both nodes of the partition. This application that runs on one of the two nodes consists of a minimal and separate instance of jboss.
My problem is the following.
When the client is under heavy traffic, it responds correctly for 7-8 hours.
After 7-8 hours it stops, waiting forever on the JNDI lookup of the Home reference of the EJB is asking for.
I appended the stacktrace of the client thread and the most of the thread dump of the server.
Can you help me ? I tried to search on forums, but I could not find anything similar.
Thanks in advance.
Ludovico
Here is the client thread:
"Thread-872" daemon prio=1 tid=0xa8743ae8 nid=0xaec sleeping[a4d8d000..a4d8d87c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0xab640e68> (a java.io.BufferedInputStream)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java
:2133)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream
.java:2423)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputSt
ream.java:2433)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1245)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324)
at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:198)
at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:11
07)
at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1192)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:514)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:507)
at javax.naming.InitialContext.lookup(InitialContext.java:347)
ce)
at ztc.tftpd.SessionControllerWrapper.handleEvent(Unknown Source)
at ztc.tftpd.ZTCWorker.getFile(Unknown Source)
at ztc.tftpd.ZTCWorker.getInputStream(Unknown Source)
at ztc.tftpd.TFTPWorker.treatReadRequest(Unknown Source)
at ztc.tftpd.TFTPWorker.processRequest(Unknown Source)
at ztc.tftpd.TFTPWorker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:534)
This is the stacktrace of the server:
"HAJNDI-AutomaticDiscovery" prio=1 tid=0x081a2290 nid=0xae6 runnable [9587f000..9587f87c]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:711)
- locked <0xa56850e8> (a java.net.DatagramPacket)
- locked <0x9e84e948> (a java.net.MulticastSocket)
at org.jboss.ha.jndi.HANamingService$AutomaticDiscovery.run(HANamingService.java:519)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x086595a8 nid=0xae6 runnable [939fb000..939fb87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e84e9a8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:372)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x08b47928 nid=0xae6 waiting for monitor entry [95e7a000..95e7a87c]
at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:917)
- waiting to lock <0x9e379258> (a sun.misc.SoftCache)
at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:396)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x0933b9d8 nid=0xae6 waiting for monitor entry [91838000..9183887c]
at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:917)
- waiting to lock <0x9e379258> (a sun.misc.SoftCache)
at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:396)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x08b4ad28 nid=0xae6 waiting for monitor entry [916b5000..916b587c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
A lot of threads as the one above are present, all waiting to lock <0x9e84f6c8>
"HAJNDI-Listener" prio=1 tid=0x08063c70 nid=0xae6 runnable [9830a000..9830a87c]
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:1647)
at java.lang.Class.getDeclaredMethod(Class.java:1254)
at java.io.ObjectOutputStream$1.run(ObjectOutputStream.java:952)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectOutputStream.auditSubclass(ObjectOutputStream.java:944)
at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:925)
- locked <0x9e379258> (a sun.misc.SoftCache)
at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
at sun.rmi.server.MarshalOutputStream.(MarshalOutputStream.java:46)
at sun.rmi.server.MarshalOutputStream.(MarshalOutputStream.java:37)
at java.rmi.MarshalledObject$MarshalledObjectOutputStream.(MarshalledObject.java:222)
at java.rmi.MarshalledObject.(MarshalledObject.java:90)
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:399)
- locked <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x087b0c50 nid=0xae6 waiting for monitor entry [9678c000..9678c87c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x088a8b10 nid=0xae6 waiting for monitor entry [9573e000..9573e87c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x086aa3c0 nid=0xae6 waiting for monitor entry [902bb000..902bb87c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x080641d8 nid=0xae6 waiting for monitor entry [94147000..9414787c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x0860d168 nid=0xae6 waiting for monitor entry [911b2000..911b287c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"HAJNDI-Listener" prio=1 tid=0x08a6c490 nid=0xae6 waiting for monitor entry [92115000..9211587c]
at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
- waiting to lock <0x9e84f6c8> (a $Proxy23)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor5" daemon prio=1 tid=0x089f83f8 nid=0xae6 in Object.wait() [8fdc8000..8fdc887c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa5179e68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0xa5179e68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
"JNP Server" prio=1 tid=0x09303078 nid=0xae6 runnable [961f7000..961f787c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e376cb8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jnp.server.Main.run(Main.java:281)
at java.lang.Thread.run(Thread.java:534)
"MERGE2.FindSubgroups thread" daemon prio=1 tid=0x08e24830 nid=0xae6 runnable [98208000..9820887c]
at org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:230)
at java.lang.Thread.run(Thread.java:534)
"ReusableThread" daemon prio=1 tid=0x08084418 nid=0xae6 in Object.wait() [944c7000..944c787c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4ef2490> (a org.jgroups.util.ReusableThread)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.ReusableThread.run(ReusableThread.java:176)
- locked <0xa4ef2490> (a org.jgroups.util.ReusableThread)
at java.lang.Thread.run(Thread.java:534)
"Scheduler main thread" daemon prio=1 tid=0x087a1ff8 nid=0xae6 in Object.wait() [9850e000..9850e87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4eefa08> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.peek(Queue.java:377)
- locked <0xa4eefa08> (a java.lang.Object)
at org.jgroups.util.Scheduler.run(Scheduler.java:64)
at java.lang.Thread.run(Thread.java:534)
"UDP mcast receiver" daemon prio=1 tid=0x0891cf40 nid=0xae6 in Object.wait() [98690000..9869187c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e24da40> (a sun.misc.SoftCache)
at java.lang.Object.wait(Object.java:429)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:862)
- locked <0x9e24da40> (a sun.misc.SoftCache)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:730)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:538)
at sun.text.resources.LocaleData$1.run(LocaleData.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at sun.text.resources.LocaleData.getBundle(LocaleData.java:111)
at sun.text.resources.LocaleData.getDateFormatZoneData(LocaleData.java:107)
at java.text.DateFormatSymbols.cacheLookup(DateFormatSymbols.java:430)
at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:468)
at java.text.DateFormatSymbols.(DateFormatSymbols.java:103)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:442)
at java.text.SimpleDateFormat.(SimpleDateFormat.java:424)
at org.jgroups.log.Format.formatTimestamp(Format.java:58)
at org.jgroups.log.Tracer.timestamp(Tracer.java:192)
at org.jgroups.log.Tracer.logString(Tracer.java:173)
at org.jgroups.log.Tracer.print(Tracer.java:123)
at org.jgroups.log.Trace.print(Trace.java:632)
at org.jgroups.log.Trace.println(Trace.java:661)
at org.jgroups.log.Trace.error(Trace.java:686)
at org.jgroups.protocols.UDP.handleIncomingUdpPacket(UDP.java:554)
at org.jgroups.protocols.UDP.run(UDP.java:210)
at java.lang.Thread.run(Thread.java:534)
"TimeScheduler.Thread" daemon prio=1 tid=0x0867cb00 nid=0xae6 in Object.wait() [98793000..9879387c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4eca3c8> (a org.jgroups.util.TimeScheduler$TaskQueue)
at org.jgroups.util.TimeScheduler._run(TimeScheduler.java:323)
- locked <0xa4eca3c8> (a org.jgroups.util.TimeScheduler$TaskQueue)
at org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:45)
at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:117)
at java.lang.Thread.run(Thread.java:534)
"UpHandler (STATE_TRANSFER)" daemon prio=1 tid=0x0867bf40 nid=0xae6 in Object.wait() [98814000..9881487c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4ede670> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4ede670> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (STATE_TRANSFER)" daemon prio=1 tid=0x0867ae80 nid=0xae6 in Object.wait() [98895000..9889587c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4ede698> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4ede698> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (GMS)" daemon prio=1 tid=0x0867a2c0 nid=0xae6 in Object.wait() [98916000..9891687c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4ede2c8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4ede2c8> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (GMS)" daemon prio=1 tid=0x08b4ab78 nid=0xae6 in Object.wait() [98997000..9899787c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4ede2f0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4ede2f0> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (FRAG)" daemon prio=1 tid=0x08b49ab8 nid=0xae6 in Object.wait() [98a18000..98a1887c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edc030> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edc030> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (FRAG)" daemon prio=1 tid=0x08b48ef8 nid=0xae6 in Object.wait() [98a99000..98a9987c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edc058> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edc058> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (UNICAST)" daemon prio=1 tid=0x08b48338 nid=0xae6 in Object.wait() [98b1a000..98b1a87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edbdf8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edbdf8> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (UNICAST)" daemon prio=1 tid=0x087b1660 nid=0xae6 in Object.wait() [98b9b000..98b9b87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edbe20> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edbe20> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (STABLE)" daemon prio=1 tid=0x087b0aa0 nid=0xae6 in Object.wait() [98c1c000..98c1c87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edbbc0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edbbc0> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (STABLE)" daemon prio=1 tid=0x08f6ad90 nid=0xae6 in Object.wait() [98c9d000..98c9d87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edbbe8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edbbe8> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (NAKACK)" daemon prio=1 tid=0x088a8d58 nid=0xae6 in Object.wait() [98d1e000..98d1e87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb900> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb900> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (NAKACK)" daemon prio=1 tid=0x08682860 nid=0xae6 in Object.wait() [98d9f000..98d9f87c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb928> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb928> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x087a41b8 nid=0xae6 in Object.wait() [98e20000..98e2087c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb708> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb708> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x0867cd20 nid=0xae6 in Object.wait() [98ea1000..98ea187c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb730> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb730> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (FD)" daemon prio=1 tid=0x0897e030 nid=0xae6 in Object.wait() [98f22000..98f2287c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb410> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb410> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (FD)" daemon prio=1 tid=0x08b6dfd8 nid=0xae6 in Object.wait() [98fa3000..98fa387c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb438> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb438> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"UpHandler (PING)" daemon prio=1 tid=0x08f6c3e8 nid=0xae6 in Object.wait() [98610000..9861087c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb080> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb080> (a java.lang.Object)
at org.jgroups.stack.UpHandler.run(Protocol.java:44)
"DownHandler (PING)" daemon prio=1 tid=0x08405490 nid=0xae6 in Object.wait() [97e00000..97e0087c]
at java.lang.Object.wait(Native Method)
- waiting on <0xa4edb0a8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jgroups.util.Queue.remove(Queue.java:217)
- locked <0xa4edb0a8> (a java.lang.Object)
at org.jgroups.stack.DownHandler.run(Protocol.java:98)
"Thread-126" daemon prio=1 tid=0x08a74020 nid=0xae6 runnable [98106000..9810687c]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <0x9ec03458> (a java.io.BufferedInputStream)
at com.sun.jndi.ldap.Connection.run(Connection.java:780)
at java.lang.Thread.run(Thread.java:534)
"IdleRemover" daemon prio=1 tid=0x08c24918 nid=0xae6 in Object.wait() [9794e000..9794e87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9ebf4890> (a java.util.ArrayList)
at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
- locked <0x9ebf4890> (a java.util.ArrayList)
at java.lang.Thread.run(Thread.java:534)
"Thread-84" prio=1 tid=0x081e6f40 nid=0xae6 in Object.wait() [9774a000..9774a87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9ebf49b8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
- locked <0x9ebf49b8> (a java.lang.Object)
"Thread-83" daemon prio=1 tid=0x08c27bb8 nid=0xae6 runnable [97afa000..97afa87c]
at java.util.TaskQueue.fixDown(Timer.java:553)
at java.util.TaskQueue.rescheduleMin(Timer.java:501)
at java.util.TimerThread.mainLoop(Timer.java:422)
- locked <0x9ebc7aa0> (a java.lang.Object)
- locked <0x9ebc7a68> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"StandardManager[/ztc]" daemon prio=1 tid=0x08c36748 nid=0xae6 waiting on condition [974ab000..974ab87c]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
at java.lang.Thread.run(Thread.java:534)
"Connection Consumer for dest TOPIC.ztcevents" prio=1 tid=0x089ff438 nid=0xae6 in Object.wait() [975c7000..975c787c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9eaa4d78> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:429)
at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
- locked <0x9eaa4d78> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:534)
"UILServerILService Accept Thread" prio=1 tid=0x08e59598 nid=0xae6 runnable [97bfc000..97bfc87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e933280> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
at java.lang.Thread.run(Thread.java:534)
"RMI TCP Accept-0" daemon prio=1 tid=0x08e59058 nid=0xae6 runnable [97c7d000..97c7d87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e92dc68> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"Thread-63" prio=1 tid=0x08904b88 nid=0xae6 in Object.wait() [97f02000..97f0287c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e91d9d0> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:429)
at org.jboss.resource.connectionmanager.PoolFiller$1.run(PoolFiller.java:75)
- locked <0x9e91d9d0> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:534)
"Thread-62" prio=1 tid=0x08905aa0 nid=0xae6 sleeping[97f83000..97f8387c]
at java.lang.Thread.sleep(Native Method)
at org.hsqldb.Log.run(Unknown Source)
at java.lang.Thread.run(Thread.java:534)
"Thread-61" daemon prio=1 tid=0x08dcf590 nid=0xae6 in Object.wait() [97e81000..97e8187c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e8ce198> (a org.jboss.util.timeout.TimeoutFactory)
at java.lang.Object.wait(Object.java:429)
at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:497)
- locked <0x9e8ce198> (a org.jboss.util.timeout.TimeoutFactory)
at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)
"Thread-60" daemon prio=1 tid=0x0857c878 nid=0xae6 waiting on condition [98004000..9800487c]
at java.util.TaskQueue.fixDown(Timer.java:553)
at java.util.TaskQueue.rescheduleMin(Timer.java:501)
at java.util.TimerThread.mainLoop(Timer.java:422)
- locked <0x9e922658> (a java.lang.Object)
- locked <0x9e8c8c58> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"RMI TCP Accept-0" daemon prio=1 tid=0x089c7c40 nid=0xae6 runnable [9838b000..9838b87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e84ea68> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"AsynchHAMembershipListener Thread" prio=1 tid=0x086812a8 nid=0xae6 in Object.wait() [9840c000..9840c87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e824058> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
- locked <0x9e824058> (a java.lang.Object)
at org.jboss.ha.framework.server.HAPartitionImpl$AsynchViewChangeHandler.run(HAPartitionImpl.java:933)
at java.lang.Thread.run(Thread.java:534)
"JBossMQ Cache Reference Softner" daemon prio=1 tid=0x08785208 nid=0xae6 runnable [99024000..9902487c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e702c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x9e702c78> (a java.lang.ref.ReferenceQueue$Lock)
at org.jboss.mq.server.MessageCache.run(MessageCache.java:225)
at java.lang.Thread.run(Thread.java:534)
"TP-Monitor" daemon prio=1 tid=0x08a7d5f0 nid=0xae6 runnable [990db000..990db87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e4c4838> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
- locked <0x9e4c4838> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor4" daemon prio=1 tid=0x08a7d2c0 nid=0xae6 runnable [9915c000..9915c87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e47ef08> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:287)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:581)
at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:767)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor3" daemon prio=1 tid=0x081e7c00 nid=0xae6 in Object.wait() [991dd000..991dd87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e47c628> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0x9e47c628> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor2" daemon prio=1 tid=0x081e7910 nid=0xae6 in Object.wait() [9925e000..9925e87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e47c6a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0x9e47c6a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor1" daemon prio=1 tid=0x081e7718 nid=0xae6 in Object.wait() [992df000..992df87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e47c728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:429)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
- locked <0x9e47c728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:534)
"TP-Monitor" daemon prio=1 tid=0x0896b010 nid=0xae6 runnable [99360000..9936087c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e47c790> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
- locked <0x9e47c790> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:534)
"Thread-4" prio=1 tid=0x08824040 nid=0xae6 runnable [998a9000..998a987c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e393438> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:211)
at java.lang.Thread.run(Thread.java:534)
"RMI TCP Accept-4444" daemon prio=1 tid=0x08785ef8 nid=0xae6 runnable [9992a000..9992a87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e38e830> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"GC Daemon" daemon prio=1 tid=0x0881bc48 nid=0xae6 runnable [99a2c000..99a2c87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e376d68> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x9e376d68> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=1 tid=0x0881ae28 nid=0xae6 in Object.wait() [99aad000..99aad87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e376db8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x9e376db8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:534)
"Thread-3" daemon prio=1 tid=0x080b15e0 nid=0xae6 in Object.wait() [99b2e000..99b2e87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e376e30> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:429)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x9e376e30> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"RMI TCP Accept-1098" daemon prio=1 tid=0x080b0fb0 nid=0xae6 runnable [99baf000..99baf87c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e376e98> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:534)
"Thread-2" daemon prio=1 tid=0x080b2738 nid=0xae6 runnable [99c30000..99c3087c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x9e376fe8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at org.jboss.web.WebServer.run(WebServer.java:258)
at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)
"Thread-0" daemon prio=1 tid=0x084822f8 nid=0xae6 runnable [99d4d000..99d4d87c]
at java.util.TaskQueue.fixDown(Timer.java:553)
at java.util.TaskQueue.rescheduleMin(Timer.java:501)
at java.util.TimerThread.mainLoop(Timer.java:422)
- locked <0x9e352608> (a java.lang.Object)
- locked <0x9e3049c8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"DestroyJavaVM" prio=1 tid=0x08052570 nid=0xae6 waiting on condition [0..bfff99a4]
"Signal Dispatcher" daemon prio=1 tid=0x080b9e78 nid=0xae6 runnable [0..0]
"Finalizer" daemon prio=1 tid=0x080b55e0 nid=0xae6 in Object.wait() [9be8f000..9be8f87c]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e24d7e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x9e24d7e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x080b5200 nid=0xae6 waiting for monitor entry [9bf10000..9bf1087c]
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:107)
- waiting to lock <0x9e24d548> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=1 tid=0x080b46f0 nid=0xae6 runnable
"VM Periodic Task Thread" prio=1 tid=0x080be330 nid=0xae6 waiting on condition
"Suspend Checker Thread" prio=1 tid=0x080b94f0 nid=0xae6 runnable