7 Replies Latest reply on Mar 9, 2008 6:17 AM by Tim Fox

    MDBs not getting messages after long run

    Panduranga Manuru Newbie

      We are running into strange problem with JBoss Messaging when we are doing performance tesing of our application after a long run (typically after 4-5 hours).
      Our application is high volume application (200-250 TPS) and during load tests everything is fine until we encounter this issue when our MDBs seem to freeze.
      When this happens the messages just pile up on the queues and MDBs do not seem to get any messages. Only way out of this sutuation is to restart the JBoss server
      -- Everything on the server seems to be fine (We can access JMX Console, resource utilization is normal etc.)
      -- There are no errors in the logs anywhere and we are sure all exceptions in application from the MDBs are handled (there are no un handled exceptions)
      -- We have taken thread dumps that seem to indicate that worker threads are waiting for work but are not given anything (though there are messages in the queue)

      We are running following configuration in our 3 node cluster
      -- JBoss 4.2.0. EAP - CP01
      -- JBoss Messaging 1.4.1 SP1
      -- Recently patched concurrent.jar to 1.34 as we found in JIRA that there are some new fixes that are missing in our JBoss package.


      Any ideas of what could be going on?

      Thread dump from JMX Console
      Total Threads: 251Total Thread Groups: 10Thread Group: system : max priority:10, demon:falseThread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, lockName:java.lang.ref.Reference$Lock@b22549java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@1255cdcjava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)Thread: Signal Dispatcher : priority:9, demon:true, threadId:5, threadState:RUNNABLE, lockName:nullThread: RMI TCP Accept-0 : priority:5, demon:true, threadId:12, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-9003 : priority:5, demon:true, threadId:13, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:21, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI Reaper : priority:5, demon:false, threadId:22, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@11af8b0java.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)java.lang.Thread.run(Thread.java:595)Thread: GC Daemon : priority:2, demon:true, threadId:23, threadState:TIMED_WAITING, lockName:sun.misc.GC$LatencyLock@27db1cjava.lang.Object.wait(Native Method)sun.misc.GC$Daemon.run(GC.java:100)Thread: RMI TCP Accept-4444 : priority:5, demon:true, threadId:32, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1101 : priority:5, demon:true, threadId:73, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-4447 : priority:5, demon:true, threadId:76, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI LeaseChecker : priority:5, demon:true, threadId:158, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)java.lang.Thread.run(Thread.java:595)Thread: RMI RenewClean-[172.31.25.56:1101] : priority:5, demon:true, threadId:182, threadState:TIMED_WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@df10ejava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)java.lang.Thread.run(Thread.java:595)Thread: RMI ConnectionExpiration-[172.31.25.56:1101] : priority:5, demon:true, threadId:3261, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)java.lang.Thread.run(Thread.java:595)Thread Group: main : max priority:10, demon:falseThread: Timer-0 : priority:5, demon:true, threadId:11, threadState:WAITING, lockName:java.util.TaskQueue@9c5dejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: DestroyJavaVM : priority:5, demon:false, threadId:15, threadState:RUNNABLE, lockName:nullThread Group: jboss : max priority:10, demon:falseThread: Timer-1 : priority:5, demon:true, threadId:17, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@fa7d0cjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: ScannerThread : priority:5, demon:true, threadId:18, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: Listener:33909 : priority:5, demon:true, threadId:25, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: Thread-4 : priority:5, demon:true, threadId:27, threadState:TIMED_WAITING, lockName:com.arjuna.ats.arjuna.coordinator.TransactionReaper@100893ajava.lang.Object.wait(Native Method)com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)Thread: Thread-5 : priority:5, demon:true, threadId:28, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)Thread: Listener:49466 : priority:5, demon:true, threadId:31, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:33, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: AcceptorThread#0:4446 : priority:5, demon:false, threadId:34, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:35, threadState:WAITING, lockName:java.lang.Object@10d43aajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:3873 : priority:5, demon:false, threadId:36, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:37, threadState:WAITING, lockName:java.lang.Object@3aa006java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:4457 : priority:5, demon:false, threadId:38, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:39, threadState:WAITING, lockName:java.lang.Object@17e27d1java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: secondaryServerSocketThread : priority:5, demon:true, threadId:40, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:889)Thread: Timer-2 : priority:5, demon:true, threadId:41, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@162f520java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:52, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:53, threadState:WAITING, lockName:java.util.LinkedList@2a2bcajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: Timer-3 : priority:5, demon:false, threadId:55, threadState:WAITING, lockName:java.util.TaskQueue@1c6dd93java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: main-FastReceiver : priority:5, demon:false, threadId:54, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:56, threadState:WAITING, lockName:java.util.LinkedList@1489fafjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:57, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpTimer : priority:5, demon:true, threadId:58, threadState:TIMED_WAITING, lockName:java.util.TreeSet@65aadfjava.lang.Object.wait(Native Method)org.opennms.protocols.snmp.SnmpTimer$TimerLogic.run(SnmpTimer.java:148)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:59, threadState:WAITING, lockName:java.util.LinkedList@4d11c9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:60, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: Timer-4 : priority:5, demon:true, threadId:61, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@111e079java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: AsynchKeyChangeHandler Thread : priority:5, demon:false, threadId:71, threadState:WAITING, lockName:java.lang.Object@6465b5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: AsynchViewChangeHandler Thread : priority:5, demon:false, threadId:72, threadState:WAITING, lockName:java.lang.Object@1d99ee0java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: PooledInvokerAcceptor#0-4448 : priority:5, demon:false, threadId:77, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: Timer-5 : priority:5, demon:true, threadId:78, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1a81e0fjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-6 : priority:5, demon:false, threadId:88, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1913aedjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-7 : priority:5, demon:true, threadId:89, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@115a52djava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-8 : priority:5, demon:false, threadId:100, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@71f2d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: RequestController-1 : priority:10, demon:true, threadId:101, threadState:WAITING, lockName:java.lang.Object@70eb8bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ServerSocketListener : priority:5, demon:true, threadId:102, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jacorb.orb.iiop.IIOPListener$Acceptor.run(Unknown Source)Thread: ORB thread : priority:5, demon:false, threadId:104, threadState:WAITING, lockName:java.lang.Object@86f3e9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.ORB.run(Unknown Source)org.jboss.iiop.CorbaORBService$1.run(CorbaORBService.java:274)java.lang.Thread.run(Thread.java:595)Thread: RequestController-2 : priority:10, demon:true, threadId:105, threadState:WAITING, lockName:java.lang.Object@167912djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-3 : priority:10, demon:true, threadId:107, threadState:WAITING, lockName:java.lang.Object@1db21bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-4 : priority:10, demon:true, threadId:108, threadState:WAITING, lockName:java.lang.Object@9350e4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-5 : priority:10, demon:true, threadId:112, threadState:WAITING, lockName:java.lang.Object@1eadcejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-6 : priority:10, demon:true, threadId:114, threadState:WAITING, lockName:java.lang.Object@cc54bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ClientMessageReceptor0 : priority:5, demon:true, threadId:116, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@ca763djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: ServerMessageReceptor0 : priority:5, demon:true, threadId:117, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@a2f12java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: RequestProcessor-1 : priority:10, demon:true, threadId:118, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@1e9d0ccjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: RequestProcessor-2 : priority:10, demon:true, threadId:119, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@18b0b4ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:120, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1549)java.lang.Thread.run(Thread.java:595)Thread: Timer-9 : priority:5, demon:true, threadId:121, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@10a9bf0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-28 : priority:5, demon:true, threadId:122, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.console.plugins.AOPLister$1.run(AOPLister.java:882)java.lang.Thread.run(Thread.java:595)Thread: IdleRemover : priority:5, demon:true, threadId:123, threadState:TIMED_WAITING, lockName:java.util.ArrayList@12e9518java.lang.Object.wait(Native Method)org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)java.lang.Thread.run(Thread.java:595)Thread: Thread-29 : priority:5, demon:true, threadId:124, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)java.lang.Thread.run(Thread.java:595)Thread: Timer-10 : priority:5, demon:true, threadId:125, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@196a9d0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-11 : priority:5, demon:true, threadId:127, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@157f1efjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Scheduler main thread : priority:5, demon:true, threadId:131, threadState:WAITING, lockName:java.lang.Object@d9e701java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.peek(Queue.java:404)org.jgroups.util.Scheduler.run(Scheduler.java:77)java.lang.Thread.run(Thread.java:595)Thread: Timer-12 : priority:5, demon:true, threadId:154, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@18a2803java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-35 : priority:5, demon:false, threadId:160, threadState:WAITING, lockName:java.lang.Object@1270bc4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Connection Consumer for dest JBossQueue[IH_Core_Request_Queue] id=1 : priority:5, demon:false, threadId:161, threadState:WAITING, lockName:java.lang.Object@275db2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:797)org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:416)org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect35.invoke(ConsumerAspect35.java)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)java.lang.Thread.run(Thread.java:595)Thread: ScannerThread : priority:5, demon:true, threadId:162, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: JBossLifeThread : priority:5, demon:false, threadId:164, threadState:WAITING, lockName:java.lang.Object@1d34f56java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)Thread: Thread-6 : priority:5, demon:false, threadId:29, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:248)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)Thread: http-172.31.25.61-8080-Acceptor-0 : priority:5, demon:true, threadId:165, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: ajp-172.31.25.61-8009-Acceptor-0 : priority:5, demon:true, threadId:166, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-1 : priority:5, demon:true, threadId:167, threadState:RUNNABLE, lockName:nullsun.management.ThreadImpl.getThreadInfo0(Native Method)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.system.server.ServerInfo.outputJdk5ThreadMXBeanInfo(ServerInfo.java:662)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:631)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.listThreadDump(ServerInfo.java:508)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)org.jboss.mx.server.Invocation.invoke(Invocation.java:86)org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)javax.servlet.http.HttpServlet.service(HttpServlet.java:710)javax.servlet.http.HttpServlet.service(HttpServlet.java:803)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-2 : priority:5, demon:true, threadId:172, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: ReusableThread : priority:5, demon:true, threadId:197, threadState:WAITING, lockName:org.jgroups.util.ReusableThread@1836b03java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.ReusableThread.run(ReusableThread.java:201)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#0[172.31.25.56:51553] : priority:5, demon:false, threadId:198, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Timer-13 : priority:5, demon:true, threadId:199, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@8bdac1java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-14 : priority:5, demon:true, threadId:200, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1d781d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-15 : priority:5, demon:true, threadId:201, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@160c68bjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: control: Socket[addr=/172.31.25.56,port=2750,localport=58074] : priority:5, demon:true, threadId:202, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)Thread: WorkerThread#1[172.31.24.54:41537] : priority:5, demon:false, threadId:225, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#2[172.31.24.54:41539] : priority:5, demon:false, threadId:226, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#3[172.31.24.54:41541] : priority:5, demon:false, threadId:227, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#4[172.31.24.54:41543] : priority:5, demon:false, threadId:228, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#5[172.31.24.54:41545] : priority:5, demon:false, threadId:233, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#6[172.31.24.54:41546] : priority:5, demon:false, threadId:234, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#7[172.31.24.54:41549] : priority:5, demon:false, threadId:235, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#8[172.31.24.54:41552] : priority:5, demon:false, threadId:236, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-57 : priority:5, demon:false, threadId:237, threadState:WAITING, lockName:java.lang.Object@14a18c3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-58 : priority:5, demon:false, threadId:238, threadState:WAITING, lockName:java.lang.Object@8d09bdjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-59 : priority:5, demon:false, threadId:239, threadState:WAITING, lockName:java.lang.Object@157feb8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-60 : priority:5, demon:false, threadId:240, threadState:WAITING, lockName:java.lang.Object@176b8d6java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#9[172.31.25.56:36526] : priority:5, demon:false, threadId:242, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-63 : priority:5, demon:false, threadId:243, threadState:WAITING, lockName:java.lang.Object@5d82a2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-64 : priority:5, demon:false, threadId:244, threadState:WAITING, lockName:java.lang.Object@1774bb5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-65 : priority:5, demon:false, threadId:245, threadState:WAITING, lockName:java.lang.Object@626e7ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-66 : priority:5, demon:false, threadId:246, threadState:WAITING, lockName:java.lang.Object@113ccc3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-67 : priority:5, demon:false, threadId:247, threadState:WAITING, lockName:java.lang.Object@101fd42java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-68 : priority:5, demon:false, threadId:248, threadState:WAITING, lockName:java.lang.Object@bb0ef5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#10[172.31.25.56:36527] : priority:5, demon:false, threadId:249, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-70 : priority:5, demon:false, threadId:250, threadState:WAITING, lockName:java.lang.Object@1b82a5fjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-71 : priority:5, demon:false, threadId:251, threadState:WAITING, lockName:java.lang.Object@14676bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-72 : priority:5, demon:false, threadId:252, threadState:WAITING, lockName:java.lang.Object@54113java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-73 : priority:5, demon:false, threadId:253, threadState:WAITING, lockName:java.lang.Object@e1a39ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-74 : priority:5, demon:false, threadId:254, threadState:WAITING, lockName:java.lang.Object@18622eajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-75 : priority:5, demon:false, threadId:256, threadState:WAITING, lockName:java.lang.Object@1de024java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Timer-16 : priority:5, demon:true, threadId:1386, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@941055java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: http-172.31.25.61-8080-3 : priority:5, demon:true, threadId:1819, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1e2b51cjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-4 : priority:5, demon:true, threadId:3114, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@b8377ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread Group: JBoss Pooled Threads : max priority:10, demon:falseThread Group: System Threads : max priority:10, demon:falseThread: JBoss System Threads(1)-1 : priority:5, demon:true, threadId:20, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.web.WebServer.run(WebServer.java:320)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-2 : priority:5, demon:true, threadId:24, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jnp.server.Main$AcceptHandler.run(Main.java:481)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-3 : priority:5, demon:true, threadId:74, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:711)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-4 : priority:5, demon:true, threadId:75, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:615)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread Group: JGroups threads : max priority:10, demon:falseThread: DiagnosticsHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:43, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:44, threadState:WAITING, lockName:java.lang.Object@1a4be49java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:45, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Tomcat-Cluster) : priority:10, demon:true, threadId:46, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:47, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:50, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:51, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DiagnosticsHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:63, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:64, threadState:WAITING, lockName:java.lang.Object@854097java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Coreservices-Production) : priority:5, demon:true, threadId:65, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Coreservices-Production) : priority:10, demon:true, threadId:66, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:67, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:70, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:80, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:81, threadState:WAITING, lockName:java.lang.Object@bf0463java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:82, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-SFSBCache) : priority:10, demon:true, threadId:83, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:84, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:87, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:91, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:92, threadState:WAITING, lockName:java.lang.Object@c5aa1djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:93, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-EntityCache) : priority:10, demon:true, threadId:94, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:95, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:98, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: FD_SOCK Ping thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:99, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:111, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DownHandler (MPING) : priority:5, demon:true, threadId:129, threadState:WAITING, lockName:java.lang.Object@1890c5ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.DownHandler.run(Protocol.java:104)Thread: UpHandler (MPING) : priority:5, demon:true, threadId:130, threadState:WAITING, lockName:java.lang.Object@e7a8b8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.UpHandler.run(Protocol.java:49)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:132, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:133, threadState:WAITING, lockName:java.lang.Object@121cf89java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:134, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=MessagingPostOffice) : priority:10, demon:true, threadId:135, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:137, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:141, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: ReceiverThread : priority:5, demon:true, threadId:142, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.MPING.run(MPING.java:263)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:144, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receiv

        • 1. Re: MDBs not getting messages after long run
          Tim Fox Master

          There's no such release as JBoss Messaging 1.4.1.SP1.

          • 2. Re: MDBs not getting messages after long run
            Panduranga Manuru Newbie

            Sorry Version we are using is 1.4.0.SP1

            • 3. Re: MDBs not getting messages after long run
              Panduranga Manuru Newbie

              We had the same problem occur few times today.
              When the problem occurred we just stopped the delivery on the MDB MBean and started delivery again. Everything started working again.

              Also when the problem occurred we took a ThreadCPUUtilization to see what threads were doing any work and found a daemon thread which was the only one using CPU cycles.
              The thread dump (before we started re-delivery of messages to MDBs again) of this thread during this period showed
              Few times we caught the thread doing this

              Thread: Thread-78 : priority:5, demon:true, threadId:328, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1000)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:574)org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:413)org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)org.jboss.remoting.Client.invoke(Client.java:1634)org.jboss.remoting.Client.invoke(Client.java:548)org.jboss.remoting.Client.invoke(Client.java:536)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:187)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:158)org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$acknowledgeDeliveries$aop(ClientSessionDelegate.java:175)org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.invokeNext(ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.java)org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.invokeNext(ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.invokeNext(ClientSessionDelegate$acknowledgeDeliveries_1767561430588382789.java)org.jboss.jms.client.delegate.ClientSessionDelegate.acknowledgeDeliveries(ClientSessionDelegate.java)org.jboss.jms.client.container.SessionAspect.acknowledgeDeliveries(SessionAspect.java:908)org.jboss.jms.client.container.SessionAspect.handleAcknowledgeAll(SessionAspect.java:399)org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect20.invoke(SessionAspect20.java)org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeAll_3428828241771477154.invokeNext(ClientSessionDelegate$acknowledgeAll_3428828241771477154.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeAll_3428828241771477154.invokeNext(ClientSessionDelegate$acknowledgeAll_3428828241771477154.java)org.jboss.jms.client.delegate.ClientSessionDelegate.acknowledgeAll(ClientSessionDelegate.java)org.jboss.jms.message.MessageProxy.acknowledge(MessageProxy.java:426)org.jboss.messaging.core.impl.clusterconnection.MessageSucker.onMessage(MessageSucker.java:300)org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:157)org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:965)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)java.lang.Thread.run(Thread.java:595)


              Once the thread was doing
              Thread: Thread-78 : priority:5, demon:true, threadId:328, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1000)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:574)org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:413)org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)org.jboss.remoting.Client.invoke(Client.java:1634)org.jboss.remoting.Client.invoke(Client.java:548)org.jboss.remoting.Client.invoke(Client.java:536)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:187)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:158)org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$cancelDelivery$aop(ClientSessionDelegate.java:493)org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$cancelDelivery_2980985335510575813.invokeNext(ClientSessionDelegate$cancelDelivery_2980985335510575813.java)org.jboss.jms.client.delegate.ClientSessionDelegate.cancelDelivery(ClientSessionDelegate.java)org.jboss.jms.client.container.ClientConsumer.checkExpiredOrReachedMaxdeliveries(ClientConsumer.java:103)org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:132)org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:965)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)java.lang.Thread.run(Thread.java:595)


              Any idea what is going on?
              We use DUPS_OK_ACKNOWLEDGE mode for the MDBs. Does this have anything to do with these or something going on with the clustering?





              • 4. Re: MDBs not getting messages after long run
                Tim Fox Master

                Does the same problem also occur with 1.4.0.SP3?

                Can you also ensure you are using JBoss Remoting 2.2.2.SP4?

                • 6. Re: MDBs not getting messages after long run
                  Panduranga Manuru Newbie

                  Strange.
                  We finally traced back the hanging/n delivery of messages to Message expiries.

                  We set out message expiries to 5 seconds and during certain times our application would be working so hard (It is processing messages at 250 TPS) that expired messages would buildup in our queues.

                  However we do not see any messages in the Expiry queue (or small no of messages)

                  At some point Container stops delivering messages altogehter resulting in pileup.
                  -- Our suspicion is Container is never getting to newly arrived messages also and hence resulting in expiry of all messages

                  It was easy to simulate once we found this to be the cause.
                  We set message expiry to 500 ms and pumped in at about 100 TPS with service processing time of about 150 ms.
                  Container delivered abuout 50 messages and not even single message after that.

                  We are putting in a workaround to set our messages to never expire.
                  However would consider this as a defect in JBM as I expect it to handle it more gracefully and clean up expired messages or ensure these do not affect new messages

                  • 7. Re: MDBs not getting messages after long run
                    Tim Fox Master

                     

                    "mpr_prabhu" wrote:

                    However would consider this as a defect in JBM as I expect it to handle it more gracefully and clean up expired messages or ensure these do not affect new messages


                    The JMS spec only requires that expired messages are never received by a consumer. It does not require that expired messages are actively expired from the queue before consumption nor does the JMS spec require expiry queues.

                    Having said that, JBM 2.0 will implement active message expiry from queues before consumption, probably with some kind of reaper thread.

                    However, even with a reaper thread, there can be no hard guarantees of timely expiry before consumption - that would be very hard to implement. The reaper thread will take a finite time to scan the queue, and during this time there would still be the possibility of expired message buildup if you send messages fast enough.

                    One thing I have thought about, is a new special queue type that may be useful with time limited messages (e.g. stock prices).

                    With the special queue type, you send the message with a special header, say, JBM_TIME_LIMITED_ID, and set the value to, say, your stick ticker code for the instrument, e.g. RHT. Then the queue itself makes sure it only maintains the latest version of message with a specific value of that header, and discards earlier ones.

                    E.g. if I send

                    Message A - MSFT price
                    Message B - MSFT price
                    Message C - RHT price
                    Message D - IBM price
                    Message E - RHT price
                    Message F - RHT price.

                    Then I start consuming from the queue I will only receive messages:

                    B, D, F - i.e. the latest prices for each stock.

                    Just an idea, but I think would be a nice feature and useful for some applications.