1 2 Previous Next 21 Replies Latest reply on Dec 22, 2008 3:52 PM by brian.stansberry

    strace shows futex

    mohitanchlia

      jboss seems to be stuck during electing master node. when I run strace it shows me:

      strace -p 4668
      Process 4668 attached - interrupt to quit
      futex(0xa79c2e04, FUTEX_WAIT, 1, NULL

      --

      when I run jstack it gives me:

      Thread 4678: (state = BLOCKED)
      Error occurred during stack walking:
      sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:134)
      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:437)
      at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:48)
      at sun.jvm.hotspot.runtime.linux_x86.LinuxX86JavaThreadPDAccess.getCurrentFrameGuess(LinuxX86JavaThreadPDAccess.java:75)
      at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)

      --

      But I don't know where it's being blocked.

        • 1. Re: strace shows futex
          brian.stansberry

          Try getting a thread dump as discussed at http://www.jboss.org/community/docs/DOC-12300; what you posted doesn't show anything that means anything to me.

          • 2. Re: strace shows futex
            mohitanchlia

            Here is the thread dump from one of the nodes:


            "ClientConnectionHandler" daemon prio=1 tid=0x080810a8 nid=0xba7 runnable [0x9f813000..0x9f813f30]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)

            "FD_SOCK Ping thread (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x0936c930 nid=0xba6 runnable [0x9e673000..0x9e673eb0]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-6" prio=1 tid=0xa4dc5460 nid=0xba1 in Object.wait() [0x9e6f5000..0x9e6f51b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb107cbb0> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0xb107cbb0> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "RMI LeaseChecker" daemon prio=1 tid=0x08a068d0 nid=0xba0 waiting on condition [0x9e776000..0x9e776130]
            at java.lang.Thread.sleep(Native Method)
            at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
            at java.lang.Thread.run(Thread.java:595)

            "ClientConnectionHandler" daemon prio=1 tid=0x08eb1008 nid=0xb9e runnable [0x9e879000..0x9e87a030]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)

            "FD_SOCK Ping thread (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0x092b7180 nid=0xb9b runnable [0x9e8fa000..0x9e8faf30]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)
            at java.lang.Thread.run(Thread.java:595)

            "ClientConnectionHandler" daemon prio=1 tid=0x092ec570 nid=0xb8a runnable [0x9ea7d000..0x9ea7de30]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)

            "FD_SOCK Ping thread (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa4e63230 nid=0xb89 runnable [0x9e9fc000..0x9e9fd1b0]
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.read(SocketInputStream.java:129)
            at java.net.SocketInputStream.read(SocketInputStream.java:182)
            at org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)
            at java.lang.Thread.run(Thread.java:595)

            "ServerSocket acceptor thread (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa4dc46f0 nid=0xb86 runnable [0x9eafe000..0x9eaff030]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb11bae90> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)
            at java.lang.Thread.run(Thread.java:595)

            "UDP mcast receiver (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa60bcfc0 nid=0xb85 runnable [0x9eb7f000..0x9eb7ffb0]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb1247400> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb1247400> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb127e940> (a java.net.DatagramPacket)
            - locked <0xb1246f10> (a java.net.MulticastSocket)
            at org.jgroups.protocols.UDP.run(UDP.java:246)
            at java.lang.Thread.run(Thread.java:595)

            "UDP.UcastReceiverThread (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa64d8160 nid=0xb84 runnable [0x9ec00000..0x9ec00f30]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb1247070> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb1247070> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb127e960> (a java.net.DatagramPacket)
            - locked <0xb1246ef0> (a java.net.DatagramSocket)
            at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)
            at java.lang.Thread.run(Thread.java:595)

            "IncomingPacketHandler (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa5366c28 nid=0xb83 in Object.wait() [0x9ec81000..0x9ec81eb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb1247530> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jgroups.util.Queue.remove(Queue.java:263)
            - locked <0xb1247530> (a java.lang.Object)
            at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)
            at java.lang.Thread.run(Thread.java:595)

            "DiagnosticsHandler (channel=EfeItPerfPartition-EfeCache)" daemon prio=1 tid=0xa5ed6b58 nid=0xb82 runnable [0x9ed02000..0x9ed02e30]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb12575e8> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb12575e8> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb12a3518> (a java.net.DatagramPacket)
            - locked <0xb12474d0> (a java.net.MulticastSocket)
            at org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-5" daemon prio=1 tid=0xa47fcab0 nid=0xb80 in Object.wait() [0x9ed83000..0x9ed84130]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb11bad88> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0xb11bad88> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "PooledInvokerAcceptor#0-4448" prio=1 tid=0xa476fb20 nid=0xb7f runnable [0x9ee04000..0x9ee050b0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb107c1b8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)
            at java.lang.Thread.run(Thread.java:595)

            "RMI TCP Accept-4447" daemon prio=1 tid=0xa4770100 nid=0xb7e runnable [0x9ee85000..0x9ee86030]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb107b9d0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

            "JBoss System Threads(1)-4" daemon prio=1 tid=0xa43d3b90 nid=0xb7d runnable [0x9ef06000..0x9ef06fb0]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb127eaa0> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb127eaa0> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb127eae0> (a java.net.DatagramPacket)
            - locked <0xb127eb00> (a java.net.MulticastSocket)
            at org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:615)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "JBoss System Threads(1)-3" daemon prio=1 tid=0xa4770338 nid=0xb7c runnable [0x9ef87000..0x9ef87f30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb12a3828> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:711)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "RMI TCP Accept-1101" daemon prio=1 tid=0xa47705c8 nid=0xb7b runnable [0x9f008000..0x9f008eb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb107b348> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

            "AsynchViewChangeHandler Thread" prio=1 tid=0xa5c0c4a8 nid=0xb7a in Object.wait() [0x9f089000..0x9f089e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb11c75f0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
            - locked <0xb11c75f0> (a java.lang.Object)
            at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
            at java.lang.Thread.run(Thread.java:595)

            "AsynchKeyChangeHandler Thread" prio=1 tid=0xa5c0c0f8 nid=0xb79 in Object.wait() [0x9f18c000..0x9f18c1b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb1084778> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
            - locked <0xb1084778> (a java.lang.Object)
            at org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
            at java.lang.Thread.run(Thread.java:595)

            "ServerSocket acceptor thread (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0xa4dff4d0 nid=0xb74 runnable [0x9f20c000..0x9f20cfb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb11d8518> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)
            at java.lang.Thread.run(Thread.java:595)

            "UDP mcast receiver (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0xa5cf6e20 nid=0xb73 runnable [0x9f28d000..0x9f28df30]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb11e7bf0> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb11e7bf0> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb127f5c8> (a java.net.DatagramPacket)
            - locked <0xb11e7708> (a java.net.MulticastSocket)
            at org.jgroups.protocols.UDP.run(UDP.java:246)
            at java.lang.Thread.run(Thread.java:595)

            "UDP.UcastReceiverThread (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0xa5e8f918 nid=0xb72 runnable [0x9f30e000..0x9f30eeb0]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb11e7868> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb11e7868> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb12a3b78> (a java.net.DatagramPacket)
            - locked <0xb11e76e8> (a java.net.DatagramSocket)
            at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)
            at java.lang.Thread.run(Thread.java:595)

            "IncomingPacketHandler (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0xa5e8f790 nid=0xb71 in Object.wait() [0x9f38f000..0x9f38fe30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb11e7d80> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jgroups.util.Queue.remove(Queue.java:263)
            - locked <0xb11e7d80> (a java.lang.Object)
            at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)
            at java.lang.Thread.run(Thread.java:595)

            "DiagnosticsHandler (channel=EfeItPerfBackEndPartition)" daemon prio=1 tid=0xa4e4b368 nid=0xb70 runnable [0x9f411000..0x9f4111b0]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb11e7f68> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb11e7f68> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb127fbd0> (a java.net.DatagramPacket)
            - locked <0xb11e7cc0> (a java.net.MulticastSocket)
            at org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-4" daemon prio=1 tid=0xa4e82a60 nid=0xb6e in Object.wait() [0x9f491000..0x9f4920b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb11e70e0> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0xb11e70e0> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "SubscriptionWatchDog" prio=1 tid=0x080f9570 nid=0xb6b waiting on condition [0xa42fe000..0xa42ff030]
            at java.lang.Thread.sleep(Native Method)
            at org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)
            at java.lang.Thread.run(Thread.java:595)

            "ServerSocket acceptor thread (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x08b02220 nid=0xb67 runnable [0xa44fe000..0xa44fee30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb0ef13a0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)
            at java.lang.Thread.run(Thread.java:595)

            "UDP mcast receiver (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x08e4dd80 nid=0xb66 runnable [0xa52ed000..0xa52ee1b0]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb0ef10d0> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb0ef10d0> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb0f571c0> (a java.net.DatagramPacket)
            - locked <0xb0ed2928> (a java.net.MulticastSocket)
            at org.jgroups.protocols.UDP.run(UDP.java:246)
            at java.lang.Thread.run(Thread.java:595)

            "UDP.UcastReceiverThread (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x08c4b750 nid=0xb65 runnable [0xa59fe000..0xa59ff130]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb0ed29d8> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb0ed29d8> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb0f25220> (a java.net.DatagramPacket)
            - locked <0xb0ed2908> (a java.net.DatagramSocket)
            at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)
            at java.lang.Thread.run(Thread.java:595)

            "IncomingPacketHandler (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x08d1e578 nid=0xb64 in Object.wait() [0xa5bfe000..0xa5bff0b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0efb4c8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jgroups.util.Queue.remove(Queue.java:263)
            - locked <0xb0efb4c8> (a java.lang.Object)
            at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)
            at java.lang.Thread.run(Thread.java:595)

            "DiagnosticsHandler (channel=Tomcat-EfeItPerfBackEndPartition)" daemon prio=1 tid=0x08086af0 nid=0xb63 runnable [0xa5dfe000..0xa5dff030]
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0xb0ef1250> (a java.net.PlainDatagramSocketImpl)
            at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
            - locked <0xb0ef1250> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:712)
            - locked <0xb0f81480> (a java.net.DatagramPacket)
            - locked <0xb0efaef0> (a java.net.MulticastSocket)
            at org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-3" daemon prio=1 tid=0xa53fada8 nid=0xb61 in Object.wait() [0xa6399000..0xa6399f30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0efaf20> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0xb0efaf20> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "Timer-2" prio=1 tid=0x08d1ebc0 nid=0xb60 in Object.wait() [0xa5f91000..0xa5f91eb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0b47438> (a java.util.TaskQueue)
            at java.lang.Object.wait(Object.java:474)
            at java.util.TimerThread.mainLoop(Timer.java:483)
            - locked <0xb0b47438> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "ServerSocketRefresh" daemon prio=1 tid=0x08d6ea18 nid=0xb0f in Object.wait() [0x9fee4000..0x9fee4e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0892658> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
            - locked <0xb0892658> (a java.lang.Object)

            "AcceptorThread#0:3873" prio=1 tid=0x08d601e8 nid=0xb0e runnable [0x9ff66000..0x9ff661b0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb08927e0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
            at java.lang.Thread.run(Thread.java:595)

            "ServerSocketRefresh" daemon prio=1 tid=0xa4355100 nid=0xb0d in Object.wait() [0xa0232000..0xa0232130]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb03bc858> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
            - locked <0xb03bc858> (a java.lang.Object)

            "AcceptorThread#0:4446" prio=1 tid=0x08cce4b8 nid=0xb0c runnable [0xa02b2000..0xa02b30b0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb0941268> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
            at java.lang.Thread.run(Thread.java:595)

            "PooledInvokerAcceptor#0-4445" prio=1 tid=0x0815adb8 nid=0xb0b runnable [0xa0333000..0xa0334030]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb09445c0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)
            at java.lang.Thread.run(Thread.java:595)

            "RMI TCP Accept-4444" daemon prio=1 tid=0xa611ad28 nid=0xb0a runnable [0xa03b4000..0xa03b4fb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb087c6d0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

            "Listener:54189" daemon prio=1 tid=0x08b16270 nid=0xb09 runnable [0xa0435000..0xa0435f30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb094b188> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)

            "Thread-5" daemon prio=1 tid=0xa4eff6e0 nid=0xb07 waiting on condition [0xa04b6000..0xa04b6e30]
            at java.lang.Thread.sleep(Native Method)
            at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)

            "Thread-4" daemon prio=1 tid=0xa43555b8 nid=0xb06 in Object.wait() [0xa0538000..0xa05381b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0942110> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
            at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)
            - locked <0xb0942110> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)

            "Listener:54186" daemon prio=1 tid=0x08b04468 nid=0xb05 runnable [0xa05b8000..0xa05b9130]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb0891be0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)

            "JBoss System Threads(1)-2" daemon prio=1 tid=0xa5ae7a68 nid=0xb04 runnable [0xa06cc000..0xa06cd0b0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb0944920> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "GC Daemon" daemon prio=1 tid=0xa5ef2188 nid=0xb03 in Object.wait() [0xa074d000..0xa074e030]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb027a6e8> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0xb027a6e8> (a sun.misc.GC$LatencyLock)

            "RMI Reaper" prio=1 tid=0xa615f8c8 nid=0xb02 in Object.wait() [0xa07ce000..0xa07cefb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb027df78> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0xb027df78> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
            at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-1" daemon prio=1 tid=0xa615f408 nid=0xb01 in Object.wait() [0xa084f000..0xa084ff30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb02769c0> (a java.util.TaskQueue)
            at java.lang.Object.wait(Object.java:474)
            at java.util.TimerThread.mainLoop(Timer.java:483)
            - locked <0xb02769c0> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "RMI TCP Accept-1098" daemon prio=1 tid=0xa615fb50 nid=0xb00 runnable [0xa08d0000..0xa08d0eb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb0274688> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

            "JBoss System Threads(1)-1" daemon prio=1 tid=0xa55e9c40 nid=0xaff runnable [0xa0951000..0xa0951e30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0xb027e008> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.web.WebServer.run(WebServer.java:320)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "ScannerThread" daemon prio=1 tid=0xa4342f40 nid=0xafe in Object.wait() [0xa09d3000..0xa09d31b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb027a7b8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:216)
            - locked <0xb027a7b8> (a java.lang.Object)

            "Timer-0" daemon prio=1 tid=0x089e01d8 nid=0xafc in Object.wait() [0xa0a7f000..0xa0a80130]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0274ac8> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0xb0274ac8> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "DestroyJavaVM" prio=1 tid=0x0805d168 nid=0xae3 waiting on condition [0x00000000..0xfeffd010]

            "main" prio=1 tid=0x0817a910 nid=0xaf5 in Object.wait() [0xa7270000..0xa72720b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xe91e7988> (a java.util.HashMap)
            at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:501)
            - locked <0xe91e7988> (a java.util.HashMap)
            at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
            at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
            at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
            at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
            at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:797)
            at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:773)
            at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.add(DistributedReplicantManagerImpl.java:405)
            at org.jboss.ha.jmx.HAServiceMBeanSupport.registerDRMListener(HAServiceMBeanSupport.java:255)
            at org.jboss.ha.jmx.HAServiceMBeanSupport.startService(HAServiceMBeanSupport.java:177)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
            at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
            at $Proxy0.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:417)
            - locked <0xb0276e10> (a org.jboss.system.ServiceController)
            at org.jboss.system.ServiceController.start(ServiceController.java:435)
            - locked <0xb0276e10> (a org.jboss.system.ServiceController)
            at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
            at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy9.deploy(Unknown Source)
            at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
            at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
            - locked <0xb030b738> (a org.jboss.deployment.scanner.URLDeploymentScanner)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
            - locked <0xb0274940> (a org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
            at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
            at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
            at $Proxy0.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:417)
            - locked <0xb0276e10> (a org.jboss.system.ServiceController)
            at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
            at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
            at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
            at $Proxy5.deploy(Unknown Source)
            at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
            at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
            at org.jboss.Main.boot(Main.java:200)
            at org.jboss.Main$1.run(Main.java:508)
            at java.lang.Thread.run(Thread.java:595)

            "Low Memory Detector" daemon prio=1 tid=0x08156c78 nid=0xaf3 runnable [0x00000000..0x00000000]

            "CompilerThread1" daemon prio=1 tid=0x081557a8 nid=0xaf2 waiting on condition [0x00000000..0xa7517158]

            "CompilerThread0" daemon prio=1 tid=0x08154700 nid=0xaf1 waiting on condition [0x00000000..0xa75980d8]

            "AdapterThread" daemon prio=1 tid=0x08153588 nid=0xaf0 waiting on condition [0x00000000..0x00000000]

            "Signal Dispatcher" daemon prio=1 tid=0x081526e8 nid=0xaef waiting on condition [0x00000000..0x00000000]

            "Finalizer" daemon prio=1 tid=0x08147ea8 nid=0xaee in Object.wait() [0xa791b000..0xa791c130]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb0281848> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0xb0281848> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

            "Reference Handler" daemon prio=1 tid=0x08147948 nid=0xaed in Object.wait() [0xa799c000..0xa799d0b0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0xb027a6d8> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:474)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0xb027a6d8> (a java.lang.ref.Reference$Lock)

            "VM Thread" prio=1 tid=0x08145408 nid=0xaec runnable

            "GC task thread#0 (ParallelGC)" prio=1 tid=0x08087488 nid=0xae4 runnable

            "GC task thread#1 (ParallelGC)" prio=1 tid=0x080880d8 nid=0xae5 runnable

            "GC task thread#2 (ParallelGC)" prio=1 tid=0x08088d10 nid=0xae6 runnable

            "GC task thread#3 (ParallelGC)" prio=1 tid=0x08089948 nid=0xae7 runnable

            "GC task thread#4 (ParallelGC)" prio=1 tid=0x0808a580 nid=0xae8 runnable

            "GC task thread#5 (ParallelGC)" prio=1 tid=0x0808b1b8 nid=0xae9 runnable

            "GC task thread#6 (ParallelGC)" prio=1 tid=0x0808bdf0 nid=0xaea runnable

            "GC task thread#7 (ParallelGC)" prio=1 tid=0x0808ca28 nid=0xaeb runnable

            "VM Periodic Task Thread" prio=1 tid=0x08158138 nid=0xaf4 waiting on condition

            367.853: [GC [PSYoungGen: 2600K->128K(38080K)] 28309K->25837K(504128K), 0.0016620 secs]
            367.855: [Full GC [PSYoungGen: 128K->0K(38080K)] [PSOldGen: 25709K->25716K(466048K)] 25837K->25716K(504128K) [PSPermGen: 23110K->23110K(65536K)], 0.2260520 secs]
            428.095: [GC [PSYoungGen: 2000K->112K(53376K)] 27717K->25828K(519424K), 0.0016420 secs]
            428.097: [Full GC [PSYoungGen: 112K->0K(53376K)] [PSOldGen: 25716K->25713K(466048K)] 25828K->25713K(519424K) [PSPermGen: 23110K->23110K(65536K)], 0.2340420 secs]
            488.348: [GC [PSYoungGen: 2436K->112K(38080K)] 28149K->25825K(504128K), 0.0017820 secs]
            488.350: [Full GC [PSYoungGen: 112K->0K(38080K)] [PSOldGen: 25713K->25719K(466048K)] 25825K->25719K(504128K) [PSPermGen: 23110K->23110K(65536K)], 0.2242820 secs]
            548.586: [GC [PSYoungGen: 2056K->112K(52096K)] 27775K->25831K(518144K), 0.0018450 secs]
            548.588: [Full GC [PSYoungGen: 112K->0K(52096K)] [PSOldGen: 25719K->25730K(466048K)] 25831K->25730K(518144K) [PSPermGen: 23117K->23117K(65536K)], 0.2247930 secs]


            ---
            Could you please help.

            • 3. Re: strace shows futex
              brian.stansberry

               

              "main" prio=1 tid=0x0817a910 nid=0xaf5 in Object.wait() [0xa7270000..0xa72720b0]
               at java.lang.Object.wait(Native Method)
               - waiting on <0xe91e7988> (a java.util.HashMap)
               at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:501)
               - locked <0xe91e7988> (a java.util.HashMap)
               at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193)
               at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:431)
               at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192)
               at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163)
               at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:797)
               at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:773)
               at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.add(DistributedReplicantManagerImpl.java:405)
               at org.jboss.ha.jmx.HAServiceMBeanSupport.registerDRMListener(HAServiceMBeanSupport.java:255)
               at org.jboss.ha.jmx.HAServiceMBeanSupport.startService(HAServiceMBeanSupport.java:177)


              Shows the thread doing a deployment blocking waiting for a response from a group RPC. That response should either come in or the thread will unblock w/in 60 secs. (The 60 secs is configured in cluster-service.xml via the MethodCallTimeout attribute).

              It's unlikely that you just got unlucky and did the stack trace while that thread was waiting a few ms for a normal response. So, most likely something on another node is delaying/preventing the response. You can take stack traces on other nodes to see if they tell you anything. In particular you want to look at the IncomingPacketHandler threads.

              All the other threads in what you posted seem ok.

              • 4. Re: strace shows futex
                mohitanchlia

                I don't know why I see so many object waits in above thread dumps. Also, as we add node to the cluster, startup terribly becomes slow. It looks like it take a very long time inintializing 10 HA SingletonControllers we have. Sometimes it just declares one of the nodes as Dead member even though all the nodes are up and no packets have been dropped.

                • 5. Re: strace shows futex
                  mohitanchlia

                  Do you think I should add VIEW_SYNC to help remerge the views? I am not sure why the startup is slow and Jboss gets stuck or sometimes it declares node dead

                  • 6. Re: strace shows futex
                    mohitanchlia

                    I also see these messages

                    2008-11-24 17:01:08,547 WARN [protocols.pbcast.GMS] - failed to collect all ACKs (2) for view MergeView::[

                    • 7. Re: strace shows futex
                      brian.stansberry

                      Threads waiting on an object are quite normal; it's the standard mechanism via which a thread that's completed it's work is unscheduled while waiting for more work. The threads in your stack trace other than "main" all look fine. The "main" thread wait is as I described above.

                      As nodes start, each of your 10 HASingletonControllers on that node sends a message to the cluster telling it it's there. The "main" thread below shows that happening.

                      Sounds like there is an issue with handling that mesage on one or more of your nodes. If you see a log message saying a node is being suspected, get a thread dump on the node that's being suspected.

                      • 8. Re: strace shows futex
                        mohitanchlia

                        I also see:

                        2008-11-24 17:08:34,990 DEBUG [jgroups.protocols.FD] - heartbeat missing from 10.10.81.92:34144 (number=0)

                        • 9. Re: strace shows futex
                          brian.stansberry

                          Mohit,

                          It turns out your employer has a support contract with Red Hat. Please use the support case opened on our Customer Service Portal to resolve the issues you are seeing. The CSP is a much better tool for handling complex operational issues like what we're discussing on this thread.

                          Thanks!

                          • 10. Re: strace shows futex
                            mohitanchlia

                            So I upgraded the jgroup.jar to 2.6 and that resolved the issue of slowness. I do have a question, while looking around I found VIEW_SYNC in jgroups that help resolve scenario where the node is taken out of service by FD. Do you think it's worth having this parameter in UDP configuration of Jgroups?

                            • 11. Re: strace shows futex
                              brian.stansberry

                              I've asked the JGroups developers to respond to this question. VIEW_SYNC is not one of the standard protocols we include in the JBoss AS channel configurations, so I'm not as familiar with all of its pros and cons as I am with most protocols, and I don't want to give you wrong information.

                              • 12. Re: strace shows futex
                                mohitanchlia

                                Thanks a lot. Jboss support group tells us that we shouldn't upgrade jgroups just like that. I think they are working on giving us the patch.

                                Please let me know about VIEW_SYNC because what scares me is that once FD drops the member it never detects that member again and VIEW_SYNC seem to be solving that problem.

                                • 13. Re: strace shows futex
                                  brian.stansberry

                                  OK, but make sure you trace down that "FD drops the member it never detects that member again" issue on the support case, as you shouldn't be experiencing that behavior.

                                  • 14. Re: strace shows futex
                                    mohitanchlia

                                    I thought that FD retries only 'x' number of times (max_tries) after that it will not try to detect the member again. Is that not correct?

                                    Below is the config, max_tries is set to 5.

                                    <UDP mcast_addr="${partition.udpGroup:228.1.2.3}"
                                    mcast_port="${jboss.hapartition.mcast_port:45566}"
                                    tos="8"
                                    ucast_recv_buf_size="20000000"
                                    ucast_send_buf_size="640000"
                                    mcast_recv_buf_size="25000000"
                                    mcast_send_buf_size="640000"
                                    loopback="false"
                                    discard_incompatible_packets="true"
                                    enable_bundling="false"
                                    max_bundle_size="64000"
                                    max_bundle_timeout="30"
                                    use_incoming_packet_handler="true"
                                    use_outgoing_packet_handler="false"
                                    ip_ttl="${jgroups.udp.ip_ttl:2}"
                                    down_thread="false" up_thread="false"/>
                                    <PING timeout="2000"
                                    down_thread="false" up_thread="false" num_initial_members="3"/>
                                    <MERGE2 max_interval="100000"
                                    down_thread="false" up_thread="false" min_interval="20000"/>
                                    <FD_SOCK down_thread="false" up_thread="false"/>
                                    <FD timeout="10000" max_tries="5" down_thread="false" up_thread="false" shun="true"/>
                                    <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false"/>
                                    <pbcast.NAKACK max_xmit_size="60000"
                                    use_mcast_xmit="false" gc_lag="0"
                                    retransmit_timeout="300,600,1200,2400,4800"
                                    down_thread="false" up_thread="false"
                                    discard_delivered_msgs="true"/>
                                    <UNICAST timeout="300,600,1200,2400,3600"
                                    down_thread="false" up_thread="false"/>
                                    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                                    down_thread="false" up_thread="false"
                                    max_bytes="400000"/>
                                    <pbcast.GMS print_local_addr="true" join_timeout="3000"
                                    down_thread="false" up_thread="false"
                                    join_retry_timeout="2000" shun="true"
                                    view_bundling="true"/>
                                    <FRAG2 frag_size="60000" down_thread="false" up_thread="false"/>
                                    <pbcast.STATE_TRANSFER down_thread="false" up_thread="false" use_flush="false"/>

                                    1 2 Previous Next