1 Reply Latest reply on Oct 7, 2004 6:45 AM by jensmo

    HAJNDI-Listener threads hang

    lbasili

      Hello,
      I am using jboss-3.2.3.
      My application is running on a partition consisting of two nodes.
      I have a client application that uses HAJNDI to balance its requests on both nodes of the partition. This application that runs on one of the two nodes consists of a minimal and separate instance of jboss.
      My problem is the following.
      When the client is under heavy traffic, it responds correctly for 7-8 hours.
      After 7-8 hours it stops, waiting forever on the JNDI lookup of the Home reference of the EJB is asking for.
      I appended the stacktrace of the client thread and the most of the thread dump of the server.
      Can you help me ? I tried to search on forums, but I could not find anything similar.
      Thanks in advance.

      Ludovico

      Here is the client thread:


      "Thread-872" daemon prio=1 tid=0xa8743ae8 nid=0xaec sleeping[a4d8d000..a4d8d87c]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
      - locked <0xab640e68> (a java.io.BufferedInputStream)
      at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java
      :2133)
      at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream
      .java:2423)
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputSt
      ream.java:2433)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1245)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:324)
      at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:198)
      at org.jnp.interfaces.NamingContext.discoverServer(NamingContext.java:11
      07)
      at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1192)
      at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:514)
      at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:507)



      at javax.naming.InitialContext.lookup(InitialContext.java:347)
      ce)
      at ztc.tftpd.SessionControllerWrapper.handleEvent(Unknown Source)
      at ztc.tftpd.ZTCWorker.getFile(Unknown Source)
      at ztc.tftpd.ZTCWorker.getInputStream(Unknown Source)
      at ztc.tftpd.TFTPWorker.treatReadRequest(Unknown Source)
      at ztc.tftpd.TFTPWorker.processRequest(Unknown Source)
      at ztc.tftpd.TFTPWorker.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:534)



      This is the stacktrace of the server:


      "HAJNDI-AutomaticDiscovery" prio=1 tid=0x081a2290 nid=0xae6 runnable [9587f000..9587f87c]
      at java.net.PlainDatagramSocketImpl.receive(Native Method)
      at java.net.DatagramSocket.receive(DatagramSocket.java:711)
      - locked <0xa56850e8> (a java.net.DatagramPacket)
      - locked <0x9e84e948> (a java.net.MulticastSocket)
      at org.jboss.ha.jndi.HANamingService$AutomaticDiscovery.run(HANamingService.java:519)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x086595a8 nid=0xae6 runnable [939fb000..939fb87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e84e9a8> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:372)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x08b47928 nid=0xae6 waiting for monitor entry [95e7a000..95e7a87c]
      at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:917)
      - waiting to lock <0x9e379258> (a sun.misc.SoftCache)
      at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:396)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x0933b9d8 nid=0xae6 waiting for monitor entry [91838000..9183887c]
      at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:917)
      - waiting to lock <0x9e379258> (a sun.misc.SoftCache)
      at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:396)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x08b4ad28 nid=0xae6 waiting for monitor entry [916b5000..916b587c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      A lot of threads as the one above are present, all waiting to lock <0x9e84f6c8>


      "HAJNDI-Listener" prio=1 tid=0x08063c70 nid=0xae6 runnable [9830a000..9830a87c]
      at java.lang.Class.getDeclaredMethods0(Native Method)
      at java.lang.Class.privateGetDeclaredMethods(Class.java:1647)
      at java.lang.Class.getDeclaredMethod(Class.java:1254)
      at java.io.ObjectOutputStream$1.run(ObjectOutputStream.java:952)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.io.ObjectOutputStream.auditSubclass(ObjectOutputStream.java:944)
      at java.io.ObjectOutputStream.verifySubclass(ObjectOutputStream.java:925)
      - locked <0x9e379258> (a sun.misc.SoftCache)
      at java.io.ObjectOutputStream.(ObjectOutputStream.java:178)
      at sun.rmi.server.MarshalOutputStream.(MarshalOutputStream.java:46)
      at sun.rmi.server.MarshalOutputStream.(MarshalOutputStream.java:37)
      at java.rmi.MarshalledObject$MarshalledObjectOutputStream.(MarshalledObject.java:222)
      at java.rmi.MarshalledObject.(MarshalledObject.java:90)
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:399)
      - locked <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x087b0c50 nid=0xae6 waiting for monitor entry [9678c000..9678c87c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x088a8b10 nid=0xae6 waiting for monitor entry [9573e000..9573e87c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x086aa3c0 nid=0xae6 waiting for monitor entry [902bb000..902bb87c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x080641d8 nid=0xae6 waiting for monitor entry [94147000..9414787c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x0860d168 nid=0xae6 waiting for monitor entry [911b2000..911b287c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "HAJNDI-Listener" prio=1 tid=0x08a6c490 nid=0xae6 waiting for monitor entry [92115000..9211587c]
      at org.jboss.ha.jndi.HANamingService.run(HANamingService.java:397)
      - waiting to lock <0x9e84f6c8> (a $Proxy23)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Processor5" daemon prio=1 tid=0x089f83f8 nid=0xae6 in Object.wait() [8fdc8000..8fdc887c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa5179e68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Object.wait(Object.java:429)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
      - locked <0xa5179e68> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "JNP Server" prio=1 tid=0x09303078 nid=0xae6 runnable [961f7000..961f787c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e376cb8> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.jnp.server.Main.run(Main.java:281)
      at java.lang.Thread.run(Thread.java:534)

      "MERGE2.FindSubgroups thread" daemon prio=1 tid=0x08e24830 nid=0xae6 runnable [98208000..9820887c]
      at org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:230)
      at java.lang.Thread.run(Thread.java:534)

      "ReusableThread" daemon prio=1 tid=0x08084418 nid=0xae6 in Object.wait() [944c7000..944c787c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4ef2490> (a org.jgroups.util.ReusableThread)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.ReusableThread.run(ReusableThread.java:176)
      - locked <0xa4ef2490> (a org.jgroups.util.ReusableThread)
      at java.lang.Thread.run(Thread.java:534)

      "Scheduler main thread" daemon prio=1 tid=0x087a1ff8 nid=0xae6 in Object.wait() [9850e000..9850e87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4eefa08> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.peek(Queue.java:377)
      - locked <0xa4eefa08> (a java.lang.Object)
      at org.jgroups.util.Scheduler.run(Scheduler.java:64)
      at java.lang.Thread.run(Thread.java:534)

      "UDP mcast receiver" daemon prio=1 tid=0x0891cf40 nid=0xae6 in Object.wait() [98690000..9869187c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e24da40> (a sun.misc.SoftCache)
      at java.lang.Object.wait(Object.java:429)
      at java.util.ResourceBundle.findBundle(ResourceBundle.java:862)
      - locked <0x9e24da40> (a sun.misc.SoftCache)
      at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:730)
      at java.util.ResourceBundle.getBundle(ResourceBundle.java:538)
      at sun.text.resources.LocaleData$1.run(LocaleData.java:113)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.text.resources.LocaleData.getBundle(LocaleData.java:111)
      at sun.text.resources.LocaleData.getDateFormatZoneData(LocaleData.java:107)
      at java.text.DateFormatSymbols.cacheLookup(DateFormatSymbols.java:430)
      at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:468)
      at java.text.DateFormatSymbols.(DateFormatSymbols.java:103)
      at java.text.SimpleDateFormat.(SimpleDateFormat.java:442)
      at java.text.SimpleDateFormat.(SimpleDateFormat.java:424)
      at org.jgroups.log.Format.formatTimestamp(Format.java:58)
      at org.jgroups.log.Tracer.timestamp(Tracer.java:192)
      at org.jgroups.log.Tracer.logString(Tracer.java:173)
      at org.jgroups.log.Tracer.print(Tracer.java:123)
      at org.jgroups.log.Trace.print(Trace.java:632)
      at org.jgroups.log.Trace.println(Trace.java:661)
      at org.jgroups.log.Trace.error(Trace.java:686)
      at org.jgroups.protocols.UDP.handleIncomingUdpPacket(UDP.java:554)
      at org.jgroups.protocols.UDP.run(UDP.java:210)
      at java.lang.Thread.run(Thread.java:534)

      "TimeScheduler.Thread" daemon prio=1 tid=0x0867cb00 nid=0xae6 in Object.wait() [98793000..9879387c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4eca3c8> (a org.jgroups.util.TimeScheduler$TaskQueue)
      at org.jgroups.util.TimeScheduler._run(TimeScheduler.java:323)
      - locked <0xa4eca3c8> (a org.jgroups.util.TimeScheduler$TaskQueue)
      at org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:45)
      at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:117)
      at java.lang.Thread.run(Thread.java:534)

      "UpHandler (STATE_TRANSFER)" daemon prio=1 tid=0x0867bf40 nid=0xae6 in Object.wait() [98814000..9881487c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4ede670> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4ede670> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (STATE_TRANSFER)" daemon prio=1 tid=0x0867ae80 nid=0xae6 in Object.wait() [98895000..9889587c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4ede698> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4ede698> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (GMS)" daemon prio=1 tid=0x0867a2c0 nid=0xae6 in Object.wait() [98916000..9891687c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4ede2c8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4ede2c8> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (GMS)" daemon prio=1 tid=0x08b4ab78 nid=0xae6 in Object.wait() [98997000..9899787c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4ede2f0> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4ede2f0> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (FRAG)" daemon prio=1 tid=0x08b49ab8 nid=0xae6 in Object.wait() [98a18000..98a1887c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edc030> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edc030> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (FRAG)" daemon prio=1 tid=0x08b48ef8 nid=0xae6 in Object.wait() [98a99000..98a9987c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edc058> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edc058> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (UNICAST)" daemon prio=1 tid=0x08b48338 nid=0xae6 in Object.wait() [98b1a000..98b1a87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edbdf8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edbdf8> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (UNICAST)" daemon prio=1 tid=0x087b1660 nid=0xae6 in Object.wait() [98b9b000..98b9b87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edbe20> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edbe20> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (STABLE)" daemon prio=1 tid=0x087b0aa0 nid=0xae6 in Object.wait() [98c1c000..98c1c87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edbbc0> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edbbc0> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (STABLE)" daemon prio=1 tid=0x08f6ad90 nid=0xae6 in Object.wait() [98c9d000..98c9d87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edbbe8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edbbe8> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (NAKACK)" daemon prio=1 tid=0x088a8d58 nid=0xae6 in Object.wait() [98d1e000..98d1e87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb900> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb900> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (NAKACK)" daemon prio=1 tid=0x08682860 nid=0xae6 in Object.wait() [98d9f000..98d9f87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb928> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb928> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x087a41b8 nid=0xae6 in Object.wait() [98e20000..98e2087c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb708> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb708> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (VERIFY_SUSPECT)" daemon prio=1 tid=0x0867cd20 nid=0xae6 in Object.wait() [98ea1000..98ea187c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb730> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb730> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (FD)" daemon prio=1 tid=0x0897e030 nid=0xae6 in Object.wait() [98f22000..98f2287c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb410> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb410> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (FD)" daemon prio=1 tid=0x08b6dfd8 nid=0xae6 in Object.wait() [98fa3000..98fa387c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb438> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb438> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "UpHandler (PING)" daemon prio=1 tid=0x08f6c3e8 nid=0xae6 in Object.wait() [98610000..9861087c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb080> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb080> (a java.lang.Object)
      at org.jgroups.stack.UpHandler.run(Protocol.java:44)

      "DownHandler (PING)" daemon prio=1 tid=0x08405490 nid=0xae6 in Object.wait() [97e00000..97e0087c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0xa4edb0a8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jgroups.util.Queue.remove(Queue.java:217)
      - locked <0xa4edb0a8> (a java.lang.Object)
      at org.jgroups.stack.DownHandler.run(Protocol.java:98)

      "Thread-126" daemon prio=1 tid=0x08a74020 nid=0xae6 runnable [98106000..9810687c]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
      - locked <0x9ec03458> (a java.io.BufferedInputStream)
      at com.sun.jndi.ldap.Connection.run(Connection.java:780)
      at java.lang.Thread.run(Thread.java:534)

      "IdleRemover" daemon prio=1 tid=0x08c24918 nid=0xae6 in Object.wait() [9794e000..9794e87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9ebf4890> (a java.util.ArrayList)
      at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
      - locked <0x9ebf4890> (a java.util.ArrayList)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-84" prio=1 tid=0x081e6f40 nid=0xae6 in Object.wait() [9774a000..9774a87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9ebf49b8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
      - locked <0x9ebf49b8> (a java.lang.Object)

      "Thread-83" daemon prio=1 tid=0x08c27bb8 nid=0xae6 runnable [97afa000..97afa87c]
      at java.util.TaskQueue.fixDown(Timer.java:553)
      at java.util.TaskQueue.rescheduleMin(Timer.java:501)
      at java.util.TimerThread.mainLoop(Timer.java:422)
      - locked <0x9ebc7aa0> (a java.lang.Object)
      - locked <0x9ebc7a68> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:382)

      "StandardManager[/ztc]" daemon prio=1 tid=0x08c36748 nid=0xae6 waiting on condition [974ab000..974ab87c]
      at java.lang.Thread.sleep(Native Method)
      at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
      at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
      at java.lang.Thread.run(Thread.java:534)

      "Connection Consumer for dest TOPIC.ztcevents" prio=1 tid=0x089ff438 nid=0xae6 in Object.wait() [975c7000..975c787c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9eaa4d78> (a java.util.LinkedList)
      at java.lang.Object.wait(Object.java:429)
      at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
      - locked <0x9eaa4d78> (a java.util.LinkedList)
      at java.lang.Thread.run(Thread.java:534)

      "UILServerILService Accept Thread" prio=1 tid=0x08e59598 nid=0xae6 runnable [97bfc000..97bfc87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e933280> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
      at java.lang.Thread.run(Thread.java:534)

      "RMI TCP Accept-0" daemon prio=1 tid=0x08e59058 nid=0xae6 runnable [97c7d000..97c7d87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e92dc68> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-63" prio=1 tid=0x08904b88 nid=0xae6 in Object.wait() [97f02000..97f0287c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e91d9d0> (a java.util.LinkedList)
      at java.lang.Object.wait(Object.java:429)
      at org.jboss.resource.connectionmanager.PoolFiller$1.run(PoolFiller.java:75)
      - locked <0x9e91d9d0> (a java.util.LinkedList)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-62" prio=1 tid=0x08905aa0 nid=0xae6 sleeping[97f83000..97f8387c]
      at java.lang.Thread.sleep(Native Method)
      at org.hsqldb.Log.run(Unknown Source)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-61" daemon prio=1 tid=0x08dcf590 nid=0xae6 in Object.wait() [97e81000..97e8187c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e8ce198> (a org.jboss.util.timeout.TimeoutFactory)
      at java.lang.Object.wait(Object.java:429)
      at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:497)
      - locked <0x9e8ce198> (a org.jboss.util.timeout.TimeoutFactory)
      at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
      at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)

      "Thread-60" daemon prio=1 tid=0x0857c878 nid=0xae6 waiting on condition [98004000..9800487c]
      at java.util.TaskQueue.fixDown(Timer.java:553)
      at java.util.TaskQueue.rescheduleMin(Timer.java:501)
      at java.util.TimerThread.mainLoop(Timer.java:422)
      - locked <0x9e922658> (a java.lang.Object)
      - locked <0x9e8c8c58> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:382)

      "RMI TCP Accept-0" daemon prio=1 tid=0x089c7c40 nid=0xae6 runnable [9838b000..9838b87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e84ea68> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
      at java.lang.Thread.run(Thread.java:534)

      "AsynchHAMembershipListener Thread" prio=1 tid=0x086812a8 nid=0xae6 in Object.wait() [9840c000..9840c87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e824058> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:429)
      at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
      - locked <0x9e824058> (a java.lang.Object)
      at org.jboss.ha.framework.server.HAPartitionImpl$AsynchViewChangeHandler.run(HAPartitionImpl.java:933)
      at java.lang.Thread.run(Thread.java:534)

      "JBossMQ Cache Reference Softner" daemon prio=1 tid=0x08785208 nid=0xae6 runnable [99024000..9902487c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e702c78> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
      - locked <0x9e702c78> (a java.lang.ref.ReferenceQueue$Lock)
      at org.jboss.mq.server.MessageCache.run(MessageCache.java:225)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Monitor" daemon prio=1 tid=0x08a7d5f0 nid=0xae6 runnable [990db000..990db87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e4c4838> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
      at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
      - locked <0x9e4c4838> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Processor4" daemon prio=1 tid=0x08a7d2c0 nid=0xae6 runnable [9915c000..9915c87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e47ef08> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:287)
      at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:581)
      at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:767)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Processor3" daemon prio=1 tid=0x081e7c00 nid=0xae6 in Object.wait() [991dd000..991dd87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e47c628> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Object.wait(Object.java:429)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
      - locked <0x9e47c628> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Processor2" daemon prio=1 tid=0x081e7910 nid=0xae6 in Object.wait() [9925e000..9925e87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e47c6a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Object.wait(Object.java:429)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
      - locked <0x9e47c6a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Processor1" daemon prio=1 tid=0x081e7718 nid=0xae6 in Object.wait() [992df000..992df87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e47c728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Object.wait(Object.java:429)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
      - locked <0x9e47c728> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "TP-Monitor" daemon prio=1 tid=0x0896b010 nid=0xae6 runnable [99360000..9936087c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e47c790> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
      at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
      - locked <0x9e47c790> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-4" prio=1 tid=0x08824040 nid=0xae6 runnable [998a9000..998a987c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e393438> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:211)
      at java.lang.Thread.run(Thread.java:534)

      "RMI TCP Accept-4444" daemon prio=1 tid=0x08785ef8 nid=0xae6 runnable [9992a000..9992a87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e38e830> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
      at java.lang.Thread.run(Thread.java:534)

      "GC Daemon" daemon prio=1 tid=0x0881bc48 nid=0xae6 runnable [99a2c000..99a2c87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e376d68> (a sun.misc.GC$LatencyLock)
      at sun.misc.GC$Daemon.run(GC.java:100)
      - locked <0x9e376d68> (a sun.misc.GC$LatencyLock)

      "RMI Reaper" prio=1 tid=0x0881ae28 nid=0xae6 in Object.wait() [99aad000..99aad87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e376db8> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
      - locked <0x9e376db8> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
      at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-3" daemon prio=1 tid=0x080b15e0 nid=0xae6 in Object.wait() [99b2e000..99b2e87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e376e30> (a java.util.TaskQueue)
      at java.lang.Object.wait(Object.java:429)
      at java.util.TimerThread.mainLoop(Timer.java:403)
      - locked <0x9e376e30> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:382)

      "RMI TCP Accept-1098" daemon prio=1 tid=0x080b0fb0 nid=0xae6 runnable [99baf000..99baf87c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e376e98> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
      at java.lang.Thread.run(Thread.java:534)

      "Thread-2" daemon prio=1 tid=0x080b2738 nid=0xae6 runnable [99c30000..99c3087c]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
      - locked <0x9e376fe8> (a java.net.PlainSocketImpl)
      at java.net.ServerSocket.implAccept(ServerSocket.java:448)
      at java.net.ServerSocket.accept(ServerSocket.java:419)
      at org.jboss.web.WebServer.run(WebServer.java:258)
      at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)

      "Thread-0" daemon prio=1 tid=0x084822f8 nid=0xae6 runnable [99d4d000..99d4d87c]
      at java.util.TaskQueue.fixDown(Timer.java:553)
      at java.util.TaskQueue.rescheduleMin(Timer.java:501)
      at java.util.TimerThread.mainLoop(Timer.java:422)
      - locked <0x9e352608> (a java.lang.Object)
      - locked <0x9e3049c8> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:382)

      "DestroyJavaVM" prio=1 tid=0x08052570 nid=0xae6 waiting on condition [0..bfff99a4]

      "Signal Dispatcher" daemon prio=1 tid=0x080b9e78 nid=0xae6 runnable [0..0]

      "Finalizer" daemon prio=1 tid=0x080b55e0 nid=0xae6 in Object.wait() [9be8f000..9be8f87c]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x9e24d7e0> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
      - locked <0x9e24d7e0> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

      "Reference Handler" daemon prio=1 tid=0x080b5200 nid=0xae6 waiting for monitor entry [9bf10000..9bf1087c]
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:107)
      - waiting to lock <0x9e24d548> (a java.lang.ref.Reference$Lock)

      "VM Thread" prio=1 tid=0x080b46f0 nid=0xae6 runnable

      "VM Periodic Task Thread" prio=1 tid=0x080be330 nid=0xae6 waiting on condition
      "Suspend Checker Thread" prio=1 tid=0x080b94f0 nid=0xae6 runnable



        • 1. Re: HAJNDI-Listener threads hang
          jensmo

          Hi.

          I'm having the same problem in our production environment. I'm not using multicast, though. This is very critical for us. I'm thinking about upgrading to newer JBoss version.... Does anyone have a fix or configuration hints?

          OS:
          Solaris 7

          JRE for JBoss App Server:
          java version "1.4.2_04"
          Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_04-b05)
          Java HotSpot(TM) Client VM (build 1.4.2_04-b05, mixed mode)

          JBoss:
          JBoss3.2.3. Cluster consists of 2 nodes (based on the all config) in same partition on same Solaris box. We don't use multicast, but hardcoded initial_hosts using TCPPING in cluster-service.xml, which I got from this forum.

          When the cluster is running under high load (getting many remote requests from our webapps and batch applications using both HA_JNDI ports), the clients are hanging and the JBoss servers seem to not accept more incoming calls. Internal EJB-> EJB calls on each node go to local JNDI port.

          Here is my JBoss stacktrace for non-applications threads:
          [java] "RMI RenewClean-[146.192.109.65:14002]" daemon prio=5 tid=0x011d4388 nid=0x2318 in Object.wait() [e9781000..e97819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf4740d38> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "RMI RenewClean-[146.192.109.65:14001]" daemon prio=5 tid=0x012480a0 nid=0x2316 in Object.wait() [e9981000..e99819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf4740e00> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-51" daemon prio=5 tid=0x00b5fde0 nid=0x303 waiting on condition [ec781000..ec7819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.SQLWrapperBase$ExternalLockThread.run(SQLWrapperBase.java:2396)

          [java] "Thread-50" daemon prio=5 tid=0x00daf970 nid=0xc4 waiting on condition [e9a81000..e9a819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.SQLWrapperBase$ExternalLockThread.run(SQLWrapperBase.java:2396)

          [java] "Thread-49" daemon prio=5 tid=0x011cfda8 nid=0xc3 waiting on condition [e9e81000..e9e819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.SQLWrapperBase$ExternalLockThread.run(SQLWrapperBase.java:2396)

          [java] "RMI RenewClean-[172.31.89.1:46795]" daemon prio=5 tid=0x017030b8 nid=0x96 in Object.wait() [e9d81000..e9d819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf4369bd0> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "MERGE2.FindSubgroups thread" daemon prio=5 tid=0x01cfb448 nid=0x89 waiting on condition [e9481000..e94819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.jgroups.util.Util.sleep(Util.java:69)
          [java] at org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:232)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ReusableThread" daemon prio=5 tid=0x0209f558 nid=0x81 in Object.wait() [e9b81000..e9b819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf4356620> (a org.jgroups.util.ReusableThread)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.ReusableThread.run(ReusableThread.java:176)
          [java] - locked <0xf4356620> (a org.jgroups.util.ReusableThread)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-48" prio=5 tid=0x002d2b78 nid=0x79 in Object.wait() [ea481000..ea4819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf431b6f0> (a java.lang.Object)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:772)
          [java] - locked <0xf431b6f0> (a java.lang.Object)

          [java] "StandardManager[/web-console]" daemon prio=5 tid=0x02067128 nid=0x78 waiting on condition [ea681000..ea6819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
          [java] at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ScannerThread" daemon prio=5 tid=0x012a53a8 nid=0x77 waiting on condition [ec281000..ec2819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:218)
          [java] at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)

          [java] "StandardManager[/jmx-console]" daemon prio=5 tid=0x01cfe6c8 nid=0x76 waiting on condition [ec381000..ec3819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
          [java] at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-47" daemon prio=5 tid=0x00d4b640 nid=0x6d waiting on condition [ea281000..ea2819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-46" daemon prio=5 tid=0x01749328 nid=0x6c waiting on condition [ea381000..ea3819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-45" daemon prio=5 tid=0x00a086e0 nid=0x67 waiting on condition [ea881000..ea8819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-44" daemon prio=5 tid=0x00a08540 nid=0x66 waiting on condition [ea981000..ea9819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-43" daemon prio=5 tid=0x00cf9be8 nid=0x65 waiting on condition [eaa81000..eaa819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-42" daemon prio=5 tid=0x01677cc8 nid=0x64 waiting on condition [eab81000..eab819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-41" daemon prio=5 tid=0x00cf9f18 nid=0x63 waiting on condition [eac81000..eac819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-40" daemon prio=5 tid=0x01677580 nid=0x62 waiting on condition [ead81000..ead819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-39" daemon prio=5 tid=0x00fd8bd0 nid=0x5f waiting on condition [eb181000..eb1819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-38" daemon prio=5 tid=0x00fd8a30 nid=0x5e waiting on condition [eb381000..eb3819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-37" daemon prio=5 tid=0x00b148e0 nid=0x5d waiting on condition [eb481000..eb4819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$DBReconnectThread.run(DBoManager.java:867)

          [java] "Thread-36" daemon prio=5 tid=0x00bb57d8 nid=0x5c waiting on condition [eb581000..eb5819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.wrapper.sql.DBoManager$CheckTimeoutThread.run(DBoManager.java:902)

          [java] "Thread-35" daemon prio=5 tid=0x01f56380 nid=0x5a waiting on condition [ec681000..ec6819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.service.monitoring.client.InfoStorage$HouseholdThread.run(InfoStorage.java:1324)

          [java] "RMI RenewClean-[172.31.89.1:14022]" daemon prio=5 tid=0x00a9fb58 nid=0x53 in Object.wait() [ef901000..ef9019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf25e7748> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TLCJanitorThread-34" daemon prio=5 tid=0x00abf1e0 nid=0x50 waiting on condition [ebf81000..ebf819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at com.telenor.ibus.system.misc.TLCacheJanitor.run(TLCacheJanitor.java:119)

          [java] "Thread-33" daemon prio=5 tid=0x00b41090 nid=0x4e in Object.wait() [eb781000..eb7819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:503)
          [java] - locked <0xf2548a78> (a org.jboss.util.timeout.TimeoutFactory)
          [java] at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
          [java] at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)

          [java] "Thread-32" daemon prio=5 tid=0x00aac428 nid=0x4a in Object.wait() [ec981000..ec9819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.util.TimerThread.mainLoop(Timer.java:429)
          [java] - locked <0xf24746a0> (a java.util.TaskQueue)
          [java] at java.util.TimerThread.run(Timer.java:382)

          [java] "HAJNDI-AutomaticDiscovery" prio=5 tid=0x007a8cd8 nid=0x49 runnable [ec181000..ec1819f0]
          [java] at java.net.PlainDatagramSocketImpl.receive(Native Method)
          [java] - waiting to lock <0xf22b6be8> (a java.net.PlainDatagramSocketImpl)
          [java] at java.net.DatagramSocket.receive(DatagramSocket.java:711)
          [java] - locked <0xf22b6c18> (a java.net.DatagramPacket)
          [java] - locked <0xf22b6c38> (a java.net.MulticastSocket)
          [java] at org.jboss.ha.jndi.HANamingService$AutomaticDiscovery.run(HANamingService.java:519)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "RMI TCP Accept-0" daemon prio=5 tid=0x00ae7fb0 nid=0x46 runnable [ec481000..ec4819f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf22b6d58> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ClusteredHTTPSessionService - CleanupDaemon" prio=5 tid=0x00ae1550 nid=0x45 in Object.wait() [ec581000..ec5819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf2295500> (a org.jboss.ha.httpsession.server.ClusteredHTTPSessionService$CleanupDaemon)
          [java] at org.jboss.ha.httpsession.server.ClusteredHTTPSessionService$CleanupDaemon.run(ClusteredHTTPSessionService.java:455)
          [java] - locked <0xf2295500> (a org.jboss.ha.httpsession.server.ClusteredHTTPSessionService$CleanupDaemon)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "AsynchHAMembershipListener Thread" prio=5 tid=0x00a4fad8 nid=0x42 in Object.wait() [ec881000..ec8819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf226a498> (a java.lang.Object)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
          [java] - locked <0xf226a498> (a java.lang.Object)
          [java] at org.jboss.ha.framework.server.HAPartitionImpl$AsynchViewChangeHandler.run(HAPartitionImpl.java:933)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ConnectionTable.Connection.HandlerThread" daemon prio=5 tid=0x007d9200 nid=0x40 runnable [eca81000..eca819f0]
          [java] at java.net.SocketInputStream.socketRead0(Native Method)
          [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
          [java] at java.net.SocketInputStream.read(SocketInputStream.java:182)
          [java] at java.io.DataInputStream.readInt(DataInputStream.java:443)
          [java] at org.jgroups.blocks.ConnectionTable$Connection.run(ConnectionTable.java:670)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ConnectionTable.Connection.HandlerThread" daemon prio=5 tid=0x00540738 nid=0x3f runnable [ecb81000..ecb819f0]
          [java] at java.net.SocketInputStream.socketRead0(Native Method)
          [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
          [java] at java.net.SocketInputStream.read(SocketInputStream.java:182)
          [java] at java.io.DataInputStream.readInt(DataInputStream.java:443)
          [java] at org.jgroups.blocks.ConnectionTable$Connection.run(ConnectionTable.java:670)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "ConnectionTable.AcceptorThread" daemon prio=5 tid=0x00544348 nid=0x3d runnable [ecd81000..ecd819f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf2273178> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at org.jgroups.blocks.ConnectionTable.run(ConnectionTable.java:313)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TimeScheduler.Thread" daemon prio=5 tid=0x005476c0 nid=0x3c runnable [ece81000..ece819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at org.jgroups.util.TimeScheduler._run(TimeScheduler.java:323)
          [java] - locked <0xf22512f8> (a org.jgroups.util.TimeScheduler$TaskQueue)
          [java] at org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:45)
          [java] at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:117)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Scheduler main thread" daemon prio=5 tid=0x008c7698 nid=0x3b in Object.wait() [ecf81000..ecf819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf2261050> (a java.lang.Object)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.peek(Queue.java:377)
          [java] - locked <0xf2261050> (a java.lang.Object)
          [java] at org.jgroups.util.Scheduler.run(Scheduler.java:64)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "UpHandler (STATE_TRANSFER)" prio=5 tid=0x008c66a0 nid=0x3a runnable [ed081000..ed0819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2261100> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (STATE_TRANSFER)" prio=5 tid=0x008c52a8 nid=0x39 in Object.wait() [ed181000..ed1819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2261180> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (GMS)" prio=5 tid=0x008c46b8 nid=0x38 in Object.wait() [ed281000..ed2819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2261200> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (GMS)" prio=5 tid=0x008c3ac8 nid=0x37 in Object.wait() [ed381000..ed3819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2268e18> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (FRAG)" prio=5 tid=0x008c2ed8 nid=0x36 in Object.wait() [ed481000..ed4819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22573f0> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (FRAG)" prio=5 tid=0x008c22e8 nid=0x35 in Object.wait() [ed581000..ed5819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257418> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (UNICAST)" prio=5 tid=0x008c16f8 nid=0x34 in Object.wait() [ed681000..ed6819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257440> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (UNICAST)" prio=5 tid=0x00ab5ff8 nid=0x33 in Object.wait() [ed781000..ed7819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257468> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (STABLE)" prio=5 tid=0x00ab5408 nid=0x32 in Object.wait() [ed881000..ed8819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257490> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (STABLE)" prio=5 tid=0x00ab4818 nid=0x31 in Object.wait() [ed981000..ed9819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22574b8> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (NAKACK)" prio=5 tid=0x00ab3c28 nid=0x30 in Object.wait() [eda81000..eda819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22574e0> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (NAKACK)" prio=5 tid=0x00aaeb98 nid=0x2f in Object.wait() [edb81000..edb819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257508> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (VERIFY_SUSPECT)" prio=5 tid=0x00aadfa8 nid=0x2e in Object.wait() [edc81000..edc819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257530> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (VERIFY_SUSPECT)" prio=5 tid=0x00a5bd70 nid=0x2d in Object.wait() [edd81000..edd819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257558> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (FD)" prio=5 tid=0x00a5c790 nid=0x2c in Object.wait() [ede81000..ede819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257580> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (FD)" prio=5 tid=0x00ab68c8 nid=0x2b in Object.wait() [edf81000..edf819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22575a8> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "UpHandler (TCPPING)" prio=5 tid=0x00ab3288 nid=0x2a in Object.wait() [ee081000..ee0819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22575d0> (a java.lang.Object)
          [java] at org.jgroups.stack.UpHandler.run(Protocol.java:44)

          [java] "DownHandler (TCPPING)" prio=5 tid=0x00ab65e0 nid=0x29 in Object.wait() [ee181000..ee1819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf22575f8> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "DownHandler (TCP)" prio=5 tid=0x00a5c150 nid=0x28 in Object.wait() [ee281000..ee2819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jgroups.util.Queue.remove(Queue.java:217)
          [java] - locked <0xf2257620> (a java.lang.Object)
          [java] at org.jgroups.stack.DownHandler.run(Protocol.java:98)

          [java] "SnmpPortal--1-FastReceiver" prio=5 tid=0x0073e3e8 nid=0x27 runnable [ee381000..ee3819f0]
          [java] at java.net.PlainDatagramSocketImpl.receive(Native Method)
          [java] at java.net.DatagramSocket.receive(DatagramSocket.java:711)
          [java] - locked <0xf4dd0050> (a java.net.DatagramPacket)
          [java] - locked <0xf218ae68> (a java.net.DatagramSocket)
          [java] at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:305)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "SnmpPortal--1" prio=5 tid=0x0073de48 nid=0x26 runnable [ee481000..ee4819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:372)
          [java] - locked <0xf218af00> (a java.util.LinkedList)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "SnmpTimer" prio=5 tid=0x006c4df0 nid=0x25 runnable [ee581000..ee5819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at org.opennms.protocols.snmp.SnmpTimer$Scheduler.run(SnmpTimer.java:208)
          [java] - locked <0xf218af70> (a java.lang.Object)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "SnmpPortal--1-FastReceiver" prio=5 tid=0x00740158 nid=0x24 runnable [ee681000..ee6819f0]
          [java] at java.net.PlainDatagramSocketImpl.receive(Native Method)
          [java] at java.net.DatagramSocket.receive(DatagramSocket.java:711)
          [java] - locked <0xf4dd0030> (a java.net.DatagramPacket)
          [java] - locked <0xf2153820> (a java.net.DatagramSocket)
          [java] at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:305)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-11" prio=5 tid=0x005f65e8 nid=0x23 in Object.wait() [ee781000..ee7819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf2147748> (a org.jboss.mx.util.RunnableScheduler)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:181)
          [java] - locked <0xf2147748> (a org.jboss.mx.util.RunnableScheduler)
          [java] at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "SnmpPortal--1" prio=5 tid=0x005f32d8 nid=0x22 runnable [efa01000..efa019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:372)
          [java] - locked <0xf2153900> (a java.util.LinkedList)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "StandardManager[/jboss-net]" daemon prio=5 tid=0x007c1c18 nid=0x21 waiting on condition [eea81000..eea819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
          [java] at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "StandardManager[/invoker]" daemon prio=5 tid=0x007dcff0 nid=0x20 waiting on condition [eeb81000..eeb819f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.apache.catalina.session.StandardManager.threadSleep(StandardManager.java:810)
          [java] at org.apache.catalina.session.StandardManager.run(StandardManager.java:869)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TP-Monitor" daemon prio=5 tid=0x008cb8e8 nid=0x1f in Object.wait() [eec81000..eec819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1f90fd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
          [java] at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
          [java] - locked <0xf1f90fd8> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TP-Processor4" daemon prio=5 tid=0x0086b5e0 nid=0x1e runnable [eed81000..eed819f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf1fe0140> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107)
          [java] at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387)
          [java] at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:569)
          [java] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TP-Processor3" daemon prio=5 tid=0x0086ac40 nid=0x1d in Object.wait() [eee81000..eee819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1f91138> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
          [java] - locked <0xf1f91138> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TP-Processor2" daemon prio=5 tid=0x0086c7c0 nid=0x1c in Object.wait() [eef81000..eef819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1f911b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
          [java] - locked <0xf1f911b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "TP-Processor1" daemon prio=5 tid=0x0086c620 nid=0x1b in Object.wait() [ef081000..ef0819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1f91238> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:653)
          [java] - locked <0xf1f91238> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-5" daemon prio=5 tid=0x00722078 nid=0x1a in Object.wait() [ef501000..ef5019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.util.TimerThread.mainLoop(Timer.java:429)
          [java] - locked <0xf1db46a8> (a java.util.TaskQueue)
          [java] at java.util.TimerThread.run(Timer.java:382)

          [java] "Thread-4" prio=5 tid=0x0060dda8 nid=0x19 runnable [ef601000..ef6019f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf1ca2bb0> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:211)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "RMI TCP Accept-14023" daemon prio=5 tid=0x0060c630 nid=0x17 runnable [ef701000..ef7019f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf1c80630> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "RMI LeaseChecker" daemon prio=5 tid=0x00614bb0 nid=0x16 waiting on condition [ef801000..ef8019f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "GC Daemon" daemon prio=2 tid=0x00159208 nid=0x12 in Object.wait() [efb01000..efb019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1c74170> (a sun.misc.GC$LatencyLock)
          [java] at sun.misc.GC$Daemon.run(GC.java:100)
          [java] - locked <0xf1c74170> (a sun.misc.GC$LatencyLock)

          [java] "RMI Reaper" prio=5 tid=0x001587f0 nid=0x11 in Object.wait() [efc01000..efc019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] - waiting on <0xf1c741c0> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf1c741c0> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
          [java] at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-3" daemon prio=5 tid=0x005e5188 nid=0x10 in Object.wait() [efd01000..efd019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.util.TimerThread.mainLoop(Timer.java:429)
          [java] - locked <0xf1c74238> (a java.util.TaskQueue)
          [java] at java.util.TimerThread.run(Timer.java:382)

          [java] "RMI TCP Accept-14022" daemon prio=5 tid=0x005e4940 nid=0xf runnable [efe01000..efe019f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf1c742a0> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
          [java] at java.lang.Thread.run(Thread.java:534)

          [java] "Thread-2" daemon prio=5 tid=0x001581b0 nid=0xe runnable [eff01000..eff019f0]
          [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
          [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
          [java] - locked <0xf1c74370> (a java.net.PlainSocketImpl)
          [java] at java.net.ServerSocket.implAccept(ServerSocket.java:448)
          [java] at java.net.ServerSocket.accept(ServerSocket.java:419)
          [java] at org.jboss.web.WebServer.run(WebServer.java:258)
          [java] at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)

          [java] "ScannerThread" daemon prio=5 tid=0x005feea8 nid=0xd waiting on condition [f0001000..f00019f0]
          [java] at java.lang.Thread.sleep(Native Method)
          [java] at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:218)
          [java] at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)

          [java] "Thread-0" daemon prio=5 tid=0x004dae90 nid=0xc in Object.wait() [f0101000..f01019f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.util.TimerThread.mainLoop(Timer.java:429)
          [java] - locked <0xf1bd9438> (a java.util.TaskQueue)
          [java] at java.util.TimerThread.run(Timer.java:382)

          [java] "DestroyJavaVM" prio=5 tid=0x0002ec48 nid=0x1 waiting on condition [0..ffbee128]

          [java] "Signal Dispatcher" daemon prio=10 tid=0x000c4c10 nid=0x8 waiting on condition [0..0]

          [java] "Finalizer" daemon prio=8 tid=0x000c0e88 nid=0x6 in Object.wait() [fc381000..fc3819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
          [java] - locked <0xf1b269a0> (a java.lang.ref.ReferenceQueue$Lock)
          [java] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
          [java] at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

          [java] "Reference Handler" daemon prio=10 tid=0x000bf528 nid=0x5 in Object.wait() [fde81000..fde819f0]
          [java] at java.lang.Object.wait(Native Method)
          [java] at java.lang.Object.wait(Object.java:429)
          [java] at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
          [java] - locked <0xf1b26758> (a java.lang.ref.Reference$Lock)

          [java] "VM Thread" prio=5 tid=0x000be6c0 nid=0x4 runnable

          [java] "VM Periodic Task Thread" prio=10 tid=0x000c7268 nid=0xa runnable
          [java] "Suspend Checker Thread" prio=10 tid=0x000c42a8 nid=0x7 runnable