MDBs not getting messages after long run
mpr_prabhu Mar 5, 2008 2:17 AMWe are running into strange problem with JBoss Messaging when we are doing performance tesing of our application after a long run (typically after 4-5 hours).
Our application is high volume application (200-250 TPS) and during load tests everything is fine until we encounter this issue when our MDBs seem to freeze.
When this happens the messages just pile up on the queues and MDBs do not seem to get any messages. Only way out of this sutuation is to restart the JBoss server
-- Everything on the server seems to be fine (We can access JMX Console, resource utilization is normal etc.)
-- There are no errors in the logs anywhere and we are sure all exceptions in application from the MDBs are handled (there are no un handled exceptions)
-- We have taken thread dumps that seem to indicate that worker threads are waiting for work but are not given anything (though there are messages in the queue)
We are running following configuration in our 3 node cluster
-- JBoss 4.2.0. EAP - CP01
-- JBoss Messaging 1.4.1 SP1
-- Recently patched concurrent.jar to 1.34 as we found in JIRA that there are some new fixes that are missing in our JBoss package.
Any ideas of what could be going on?
Thread dump from JMX Console
Total Threads: 251Total Thread Groups: 10Thread Group: system : max priority:10, demon:falseThread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, lockName:java.lang.ref.Reference$Lock@b22549java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@1255cdcjava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)Thread: Signal Dispatcher : priority:9, demon:true, threadId:5, threadState:RUNNABLE, lockName:nullThread: RMI TCP Accept-0 : priority:5, demon:true, threadId:12, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-9003 : priority:5, demon:true, threadId:13, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:21, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI Reaper : priority:5, demon:false, threadId:22, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@11af8b0java.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)java.lang.Thread.run(Thread.java:595)Thread: GC Daemon : priority:2, demon:true, threadId:23, threadState:TIMED_WAITING, lockName:sun.misc.GC$LatencyLock@27db1cjava.lang.Object.wait(Native Method)sun.misc.GC$Daemon.run(GC.java:100)Thread: RMI TCP Accept-4444 : priority:5, demon:true, threadId:32, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1101 : priority:5, demon:true, threadId:73, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-4447 : priority:5, demon:true, threadId:76, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI LeaseChecker : priority:5, demon:true, threadId:158, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)java.lang.Thread.run(Thread.java:595)Thread: RMI RenewClean-[172.31.25.56:1101] : priority:5, demon:true, threadId:182, threadState:TIMED_WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@df10ejava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)java.lang.Thread.run(Thread.java:595)Thread: RMI ConnectionExpiration-[172.31.25.56:1101] : priority:5, demon:true, threadId:3261, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)java.lang.Thread.run(Thread.java:595)Thread Group: main : max priority:10, demon:falseThread: Timer-0 : priority:5, demon:true, threadId:11, threadState:WAITING, lockName:java.util.TaskQueue@9c5dejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: DestroyJavaVM : priority:5, demon:false, threadId:15, threadState:RUNNABLE, lockName:nullThread Group: jboss : max priority:10, demon:falseThread: Timer-1 : priority:5, demon:true, threadId:17, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@fa7d0cjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: ScannerThread : priority:5, demon:true, threadId:18, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: Listener:33909 : priority:5, demon:true, threadId:25, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: Thread-4 : priority:5, demon:true, threadId:27, threadState:TIMED_WAITING, lockName:com.arjuna.ats.arjuna.coordinator.TransactionReaper@100893ajava.lang.Object.wait(Native Method)com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)Thread: Thread-5 : priority:5, demon:true, threadId:28, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)Thread: Listener:49466 : priority:5, demon:true, threadId:31, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:33, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: AcceptorThread#0:4446 : priority:5, demon:false, threadId:34, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:35, threadState:WAITING, lockName:java.lang.Object@10d43aajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:3873 : priority:5, demon:false, threadId:36, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:37, threadState:WAITING, lockName:java.lang.Object@3aa006java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:4457 : priority:5, demon:false, threadId:38, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:39, threadState:WAITING, lockName:java.lang.Object@17e27d1java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: secondaryServerSocketThread : priority:5, demon:true, threadId:40, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:889)Thread: Timer-2 : priority:5, demon:true, threadId:41, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@162f520java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:52, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:53, threadState:WAITING, lockName:java.util.LinkedList@2a2bcajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: Timer-3 : priority:5, demon:false, threadId:55, threadState:WAITING, lockName:java.util.TaskQueue@1c6dd93java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: main-FastReceiver : priority:5, demon:false, threadId:54, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:56, threadState:WAITING, lockName:java.util.LinkedList@1489fafjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:57, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpTimer : priority:5, demon:true, threadId:58, threadState:TIMED_WAITING, lockName:java.util.TreeSet@65aadfjava.lang.Object.wait(Native Method)org.opennms.protocols.snmp.SnmpTimer$TimerLogic.run(SnmpTimer.java:148)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:59, threadState:WAITING, lockName:java.util.LinkedList@4d11c9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:60, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: Timer-4 : priority:5, demon:true, threadId:61, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@111e079java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: AsynchKeyChangeHandler Thread : priority:5, demon:false, threadId:71, threadState:WAITING, lockName:java.lang.Object@6465b5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: AsynchViewChangeHandler Thread : priority:5, demon:false, threadId:72, threadState:WAITING, lockName:java.lang.Object@1d99ee0java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: PooledInvokerAcceptor#0-4448 : priority:5, demon:false, threadId:77, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: Timer-5 : priority:5, demon:true, threadId:78, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1a81e0fjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-6 : priority:5, demon:false, threadId:88, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1913aedjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-7 : priority:5, demon:true, threadId:89, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@115a52djava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-8 : priority:5, demon:false, threadId:100, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@71f2d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: RequestController-1 : priority:10, demon:true, threadId:101, threadState:WAITING, lockName:java.lang.Object@70eb8bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ServerSocketListener : priority:5, demon:true, threadId:102, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jacorb.orb.iiop.IIOPListener$Acceptor.run(Unknown Source)Thread: ORB thread : priority:5, demon:false, threadId:104, threadState:WAITING, lockName:java.lang.Object@86f3e9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.ORB.run(Unknown Source)org.jboss.iiop.CorbaORBService$1.run(CorbaORBService.java:274)java.lang.Thread.run(Thread.java:595)Thread: RequestController-2 : priority:10, demon:true, threadId:105, threadState:WAITING, lockName:java.lang.Object@167912djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-3 : priority:10, demon:true, threadId:107, threadState:WAITING, lockName:java.lang.Object@1db21bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-4 : priority:10, demon:true, threadId:108, threadState:WAITING, lockName:java.lang.Object@9350e4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-5 : priority:10, demon:true, threadId:112, threadState:WAITING, lockName:java.lang.Object@1eadcejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-6 : priority:10, demon:true, threadId:114, threadState:WAITING, lockName:java.lang.Object@cc54bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ClientMessageReceptor0 : priority:5, demon:true, threadId:116, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@ca763djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: ServerMessageReceptor0 : priority:5, demon:true, threadId:117, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@a2f12java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: RequestProcessor-1 : priority:10, demon:true, threadId:118, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@1e9d0ccjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: RequestProcessor-2 : priority:10, demon:true, threadId:119, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@18b0b4ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:120, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1549)java.lang.Thread.run(Thread.java:595)Thread: Timer-9 : priority:5, demon:true, threadId:121, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@10a9bf0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-28 : priority:5, demon:true, threadId:122, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.console.plugins.AOPLister$1.run(AOPLister.java:882)java.lang.Thread.run(Thread.java:595)Thread: IdleRemover : priority:5, demon:true, threadId:123, threadState:TIMED_WAITING, lockName:java.util.ArrayList@12e9518java.lang.Object.wait(Native Method)org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)java.lang.Thread.run(Thread.java:595)Thread: Thread-29 : priority:5, demon:true, threadId:124, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)java.lang.Thread.run(Thread.java:595)Thread: Timer-10 : priority:5, demon:true, threadId:125, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@196a9d0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-11 : priority:5, demon:true, threadId:127, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@157f1efjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Scheduler main thread : priority:5, demon:true, threadId:131, threadState:WAITING, lockName:java.lang.Object@d9e701java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.peek(Queue.java:404)org.jgroups.util.Scheduler.run(Scheduler.java:77)java.lang.Thread.run(Thread.java:595)Thread: Timer-12 : priority:5, demon:true, threadId:154, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@18a2803java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-35 : priority:5, demon:false, threadId:160, threadState:WAITING, lockName:java.lang.Object@1270bc4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Connection Consumer for dest JBossQueue[IH_Core_Request_Queue] id=1 : priority:5, demon:false, threadId:161, threadState:WAITING, lockName:java.lang.Object@275db2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:797)org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:416)org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect35.invoke(ConsumerAspect35.java)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)java.lang.Thread.run(Thread.java:595)Thread: ScannerThread : priority:5, demon:true, threadId:162, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: JBossLifeThread : priority:5, demon:false, threadId:164, threadState:WAITING, lockName:java.lang.Object@1d34f56java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)Thread: Thread-6 : priority:5, demon:false, threadId:29, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:248)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)Thread: http-172.31.25.61-8080-Acceptor-0 : priority:5, demon:true, threadId:165, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: ajp-172.31.25.61-8009-Acceptor-0 : priority:5, demon:true, threadId:166, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-1 : priority:5, demon:true, threadId:167, threadState:RUNNABLE, lockName:nullsun.management.ThreadImpl.getThreadInfo0(Native Method)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.system.server.ServerInfo.outputJdk5ThreadMXBeanInfo(ServerInfo.java:662)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:631)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.listThreadDump(ServerInfo.java:508)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)org.jboss.mx.server.Invocation.invoke(Invocation.java:86)org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)javax.servlet.http.HttpServlet.service(HttpServlet.java:710)javax.servlet.http.HttpServlet.service(HttpServlet.java:803)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-2 : priority:5, demon:true, threadId:172, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: ReusableThread : priority:5, demon:true, threadId:197, threadState:WAITING, lockName:org.jgroups.util.ReusableThread@1836b03java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.ReusableThread.run(ReusableThread.java:201)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#0[172.31.25.56:51553] : priority:5, demon:false, threadId:198, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Timer-13 : priority:5, demon:true, threadId:199, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@8bdac1java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-14 : priority:5, demon:true, threadId:200, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1d781d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-15 : priority:5, demon:true, threadId:201, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@160c68bjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: control: Socket[addr=/172.31.25.56,port=2750,localport=58074] : priority:5, demon:true, threadId:202, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)Thread: WorkerThread#1[172.31.24.54:41537] : priority:5, demon:false, threadId:225, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#2[172.31.24.54:41539] : priority:5, demon:false, threadId:226, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#3[172.31.24.54:41541] : priority:5, demon:false, threadId:227, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#4[172.31.24.54:41543] : priority:5, demon:false, threadId:228, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#5[172.31.24.54:41545] : priority:5, demon:false, threadId:233, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#6[172.31.24.54:41546] : priority:5, demon:false, threadId:234, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#7[172.31.24.54:41549] : priority:5, demon:false, threadId:235, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#8[172.31.24.54:41552] : priority:5, demon:false, threadId:236, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-57 : priority:5, demon:false, threadId:237, threadState:WAITING, lockName:java.lang.Object@14a18c3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-58 : priority:5, demon:false, threadId:238, threadState:WAITING, lockName:java.lang.Object@8d09bdjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-59 : priority:5, demon:false, threadId:239, threadState:WAITING, lockName:java.lang.Object@157feb8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-60 : priority:5, demon:false, threadId:240, threadState:WAITING, lockName:java.lang.Object@176b8d6java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#9[172.31.25.56:36526] : priority:5, demon:false, threadId:242, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-63 : priority:5, demon:false, threadId:243, threadState:WAITING, lockName:java.lang.Object@5d82a2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-64 : priority:5, demon:false, threadId:244, threadState:WAITING, lockName:java.lang.Object@1774bb5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-65 : priority:5, demon:false, threadId:245, threadState:WAITING, lockName:java.lang.Object@626e7ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-66 : priority:5, demon:false, threadId:246, threadState:WAITING, lockName:java.lang.Object@113ccc3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-67 : priority:5, demon:false, threadId:247, threadState:WAITING, lockName:java.lang.Object@101fd42java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-68 : priority:5, demon:false, threadId:248, threadState:WAITING, lockName:java.lang.Object@bb0ef5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#10[172.31.25.56:36527] : priority:5, demon:false, threadId:249, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-70 : priority:5, demon:false, threadId:250, threadState:WAITING, lockName:java.lang.Object@1b82a5fjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-71 : priority:5, demon:false, threadId:251, threadState:WAITING, lockName:java.lang.Object@14676bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-72 : priority:5, demon:false, threadId:252, threadState:WAITING, lockName:java.lang.Object@54113java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-73 : priority:5, demon:false, threadId:253, threadState:WAITING, lockName:java.lang.Object@e1a39ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-74 : priority:5, demon:false, threadId:254, threadState:WAITING, lockName:java.lang.Object@18622eajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-75 : priority:5, demon:false, threadId:256, threadState:WAITING, lockName:java.lang.Object@1de024java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Timer-16 : priority:5, demon:true, threadId:1386, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@941055java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: http-172.31.25.61-8080-3 : priority:5, demon:true, threadId:1819, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1e2b51cjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-4 : priority:5, demon:true, threadId:3114, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@b8377ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread Group: JBoss Pooled Threads : max priority:10, demon:falseThread Group: System Threads : max priority:10, demon:falseThread: JBoss System Threads(1)-1 : priority:5, demon:true, threadId:20, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.web.WebServer.run(WebServer.java:320)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-2 : priority:5, demon:true, threadId:24, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jnp.server.Main$AcceptHandler.run(Main.java:481)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-3 : priority:5, demon:true, threadId:74, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:711)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-4 : priority:5, demon:true, threadId:75, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:615)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread Group: JGroups threads : max priority:10, demon:falseThread: DiagnosticsHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:43, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:44, threadState:WAITING, lockName:java.lang.Object@1a4be49java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:45, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Tomcat-Cluster) : priority:10, demon:true, threadId:46, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:47, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:50, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:51, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DiagnosticsHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:63, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:64, threadState:WAITING, lockName:java.lang.Object@854097java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Coreservices-Production) : priority:5, demon:true, threadId:65, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Coreservices-Production) : priority:10, demon:true, threadId:66, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:67, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:70, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:80, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:81, threadState:WAITING, lockName:java.lang.Object@bf0463java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:82, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-SFSBCache) : priority:10, demon:true, threadId:83, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:84, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:87, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:91, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:92, threadState:WAITING, lockName:java.lang.Object@c5aa1djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:93, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-EntityCache) : priority:10, demon:true, threadId:94, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:95, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:98, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: FD_SOCK Ping thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:99, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:111, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DownHandler (MPING) : priority:5, demon:true, threadId:129, threadState:WAITING, lockName:java.lang.Object@1890c5ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.DownHandler.run(Protocol.java:104)Thread: UpHandler (MPING) : priority:5, demon:true, threadId:130, threadState:WAITING, lockName:java.lang.Object@e7a8b8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.UpHandler.run(Protocol.java:49)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:132, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:133, threadState:WAITING, lockName:java.lang.Object@121cf89java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:134, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=MessagingPostOffice) : priority:10, demon:true, threadId:135, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:137, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:141, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: ReceiverThread : priority:5, demon:true, threadId:142, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.MPING.run(MPING.java:263)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:144, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receiv