1 2 Previous Next 22 Replies Latest reply on Jul 9, 2007 4:52 PM by javajedi

    EJB3/Socket invoker - connection timeouts

      We're running into a problem with a brand new production system that has a single remote EJB. We are using JBoss EJB3, which uses JBoss Remoting to do the remote communication. We are using the standard socket invoker to handle the communication. We are using JBoss Serialization instead of Java Serialization, which, in tests, provided significantly better overall performance. We've been testing this in a stage environment for several months, with no problems. Once we got it into production, we started to hit a wall. We reach a point where the socket connections between the client and the server just start timing out. A thread dump on the client shows a bunch of threads waiting for a socket connection to the server. A thread dump on the server shows several SeverThreads waiting to be used (via ServerThread.wakeup()). The problem appears to be that the thread in SocketServerInvoker that is responsible for the calls to serverSocket.accept() is getting stuck, or at least severely slowed. In the server side thread dump, here is the stack of that accept thread:

      Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:RUNNABLE, threadLockName:null
      java.net.SocketInputStream.socketRead0(Native Method)
      java.net.SocketInputStream.read(SocketInputStream.java:129)
      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      java.io.FilterInputStream.read(FilterInputStream.java:90)
      org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
      org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
      org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:83)
      org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
      org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
      org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
      org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
      org.jboss.remoting.transport.socket.ServerSocketWrapper.(ServerSocketWrapper.java:50)
      sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
      org.jboss.remoting.transport.socket.ServerThread.(ServerThread.java:87)
      org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:429)
      org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
      java.lang.Thread.run(Thread.java:619)

      It appears that this thread, which ought to be quickly finding a ServerThread to handle the incoming request, calling ServerThread.start(), and getting back to the call to serverSocket.accept() as quickly as possible, is instead doing some socket I/O. ServerSocketWrapper.java has a method called createInputStream() that creates a new ObjectInputStream. The stack trace shows that the JBossObjectInputStream constructor is reading some bytes from the input stream, but I don't think that this problem is specific to JBoss Serialization, as java.io.ObjectInputStream's constructor also reads bytes from the stream.

      It seems to me that this could be the cause of the problem that we are seeing. Any time that you're doing a blocking socket read like this, you can potentially just sit for a while until the socket times out. When you've got a thread calling serverSocket.accept(), you can't normally afford to do something like this on that thread.

      Does this analysis make sense? Any thoughts/suggestions would be greatly appreciated.

      Thanks.

      --Tim

        • 1. Re: EJB3/Socket invoker - connection timeouts

          BTW, here are the thread dumps from the server and the client once the system is just "stuck" getting socket timeouts galore on the client side.

          SERVER:

          Total Threads: 126
          Total Thread Groups: 8
          
          Thread Group: system : max priority:10, demon:false
          
           Thread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, threadLockName:java.lang.ref.Reference$Lock@d6883
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
          
           Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@18caee7
          
           java.lang.Object.wait(Native Method)
           java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
           java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
           java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
          
           Thread: Signal Dispatcher : priority:9, demon:true, threadId:4, threadState:RUNNABLE, threadLockName:null
           Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:16, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI Reaper : priority:5, demon:false, threadId:17, threadState:WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@c21a68
          
           java.lang.Object.wait(Native Method)
           java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
           java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
           sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: GC Daemon : priority:2, demon:true, threadId:18, threadState:TIMED_WAITING, threadLockName:sun.misc.GC$LatencyLock@16ed0f7
          
           java.lang.Object.wait(Native Method)
           sun.misc.GC$Daemon.run(GC.java:100)
          
           Thread: RMI TCP Accept-4444 : priority:5, demon:true, threadId:21, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI Scheduler(0) : priority:5, demon:true, threadId:27, threadState:TIMED_WAITING, threadLockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ac0a60
          
           sun.misc.Unsafe.park(Native Method)
           java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
           java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1927)
           java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
           java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
           java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
           java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Accept-1101 : priority:5, demon:true, threadId:85, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Accept-4447 : priority:5, demon:true, threadId:88, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
           sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI RenewClean-[68.142.219.229:1101] : priority:5, demon:true, threadId:518, threadState:TIMED_WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@1e70e12
          
           java.lang.Object.wait(Native Method)
           java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
           sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
           java.lang.Thread.run(Thread.java:619)
          
          
           Thread Group: main : max priority:10, demon:false
          
           Thread: DestroyJavaVM : priority:5, demon:false, threadId:9, threadState:RUNNABLE, threadLockName:null
          
           Thread Group: jboss : max priority:10, demon:false
          
           Thread: Timer-0 : priority:5, demon:true, threadId:12, threadState:TIMED_WAITING, threadLockName:java.util.TaskQueue@1af67f0
          
           java.lang.Object.wait(Native Method)
           java.util.TimerThread.mainLoop(Timer.java:509)
           java.util.TimerThread.run(Timer.java:462)
          
           Thread: ScannerThread : priority:5, demon:true, threadId:13, threadState:TIMED_WAITING, threadLockName:null
          
           java.lang.Thread.sleep(Native Method)
           org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)
           org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
          
           Thread: TimeoutFactory-0 : priority:5, demon:true, threadId:20, threadState:WAITING, threadLockName:java.lang.Object@55ff4
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.util.timeout.TimeoutPriorityQueueImpl.poll(TimeoutPriorityQueueImpl.java:253)
           org.jboss.util.timeout.TimeoutPriorityQueueImpl.take(TimeoutPriorityQueueImpl.java:230)
           org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:215)
           org.jboss.util.timeout.TimeoutFactory.access$000(TimeoutFactory.java:41)
           org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:136)
          
           Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:22, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:261)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
           java.io.BufferedInputStream.read(BufferedInputStream.java:317)
           java.io.FilterInputStream.read(FilterInputStream.java:90)
           org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
           org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
           org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:83)
           org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
           org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
           org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
           org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
           org.jboss.remoting.transport.socket.ServerSocketWrapper.(ServerSocketWrapper.java:50)
           sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
           sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
           java.lang.reflect.Constructor.newInstance(Constructor.java:513)
           org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
           org.jboss.remoting.transport.socket.ServerThread.(ServerThread.java:87)
           org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:429)
           org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:28, threadState:TIMED_WAITING, threadLockName:null
          
           java.lang.Thread.sleep(Native Method)
           org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:29, threadState:TIMED_WAITING, threadLockName:null
          
           java.lang.Thread.sleep(Native Method)
           org.jboss.ws.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:463)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: DownHandler (TCP) : priority:5, demon:true, threadId:30, threadState:WAITING, threadLockName:java.lang.Object@16fb9fd
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: DownHandler (TCPPING) : priority:5, demon:true, threadId:31, threadState:WAITING, threadLockName:java.lang.Object@5de82a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: UpHandler (TCPPING) : priority:5, demon:true, threadId:32, threadState:WAITING, threadLockName:java.lang.Object@1d77d9e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.UpHandler.run(Protocol.java:48)
          
           Thread: DownHandler (FD) : priority:5, demon:true, threadId:33, threadState:WAITING, threadLockName:java.lang.Object@1da3acc
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: UpHandler (FD) : priority:5, demon:true, threadId:34, threadState:WAITING, threadLockName:java.lang.Object@e74663
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.UpHandler.run(Protocol.java:48)
          
           Thread: DownHandler (NAKACK) : priority:5, demon:true, threadId:35, threadState:WAITING, threadLockName:java.lang.Object@c1abcf
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: UpHandler (NAKACK) : priority:5, demon:true, threadId:36, threadState:WAITING, threadLockName:java.lang.Object@54864c
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.UpHandler.run(Protocol.java:48)
          
           Thread: DownHandler (GMS) : priority:5, demon:true, threadId:37, threadState:WAITING, threadLockName:java.lang.Object@25a5d2
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: UpHandler (GMS) : priority:5, demon:true, threadId:38, threadState:WAITING, threadLockName:java.lang.Object@1d694da
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.UpHandler.run(Protocol.java:48)
          
           Thread: DownHandler (STATE_TRANSFER) : priority:5, demon:true, threadId:39, threadState:WAITING, threadLockName:java.lang.Object@1d6e4a4
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.DownHandler.run(Protocol.java:105)
          
           Thread: UpHandler (STATE_TRANSFER) : priority:5, demon:true, threadId:40, threadState:WAITING, threadLockName:java.lang.Object@54c2b8
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.stack.UpHandler.run(Protocol.java:48)
          
           Thread: MessageDispatcher up processing thread : priority:5, demon:true, threadId:41, threadState:WAITING, threadLockName:java.lang.Object@4a0a42
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jgroups.util.Queue.remove(Queue.java:221)
           org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:689)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: SocketServerInvokerThread-68.142.216.101-1 : priority:5, demon:false, threadId:44, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1bd9de3
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-2 : priority:5, demon:false, threadId:45, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@2153fe
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-3 : priority:5, demon:false, threadId:46, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@17c7988
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-4 : priority:5, demon:false, threadId:47, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@cc7f9e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-5 : priority:5, demon:false, threadId:48, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a2da17
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-6 : priority:5, demon:false, threadId:49, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@174f02c
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-7 : priority:5, demon:false, threadId:50, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@78529d
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-8 : priority:5, demon:false, threadId:51, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@c19fbf
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-9 : priority:5, demon:false, threadId:52, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@63e563
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-10 : priority:5, demon:false, threadId:53, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@82e4f3
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-11 : priority:5, demon:false, threadId:54, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@12f9924
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-12 : priority:5, demon:false, threadId:55, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@14b74a7
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-13 : priority:5, demon:false, threadId:56, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@156d7c8
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-14 : priority:5, demon:false, threadId:57, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1364dcb
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-15 : priority:5, demon:false, threadId:58, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@103d246
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-16 : priority:5, demon:false, threadId:59, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@d0deb5
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-17 : priority:5, demon:false, threadId:60, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@136c55
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-18 : priority:5, demon:false, threadId:61, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13af084
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-19 : priority:5, demon:false, threadId:62, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@14596d5
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-20 : priority:5, demon:false, threadId:63, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@45ce17
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-21 : priority:5, demon:false, threadId:64, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@15a4523
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-22 : priority:5, demon:false, threadId:65, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13c296b
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-23 : priority:5, demon:false, threadId:66, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@fe2558
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-24 : priority:5, demon:false, threadId:67, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@24c672
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-25 : priority:5, demon:false, threadId:68, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@19bd735
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-26 : priority:5, demon:false, threadId:69, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1fa490e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-27 : priority:5, demon:false, threadId:70, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@b6a6d6
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-28 : priority:5, demon:false, threadId:71, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@19299f5
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-29 : priority:5, demon:false, threadId:72, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@10f34cc
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-30 : priority:5, demon:false, threadId:73, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4e194f
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-31 : priority:5, demon:false, threadId:74, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1d592a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-32 : priority:5, demon:false, threadId:75, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@6b51d8
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-33 : priority:5, demon:false, threadId:76, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a02097
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-34 : priority:5, demon:false, threadId:77, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4ac866
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-35 : priority:5, demon:false, threadId:78, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@3dfcb
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-36 : priority:5, demon:false, threadId:79, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@11b1d80
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: SocketServerInvokerThread-68.142.216.101-37 : priority:5, demon:false, threadId:80, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13136e5
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
          
           Thread: AsynchKeyChangeHandler Thread : priority:5, demon:false, threadId:82, threadState:WAITING, threadLockName:java.lang.Object@17ecdf0
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: AsynchViewChangeHandler Thread : priority:5, demon:false, threadId:83, threadState:WAITING, threadLockName:java.lang.Object@11cbec9
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
           org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: MERGE2.FindSubgroups thread : priority:5, demon:true, threadId:84, threadState:TIMED_WAITING, threadLockName:null
          
           java.lang.Thread.sleep(Native Method)
           org.jgroups.util.Util.sleep(Util.java:76)
           org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:233)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: PooledInvokerAcceptor#0-4446 : priority:5, demon:false, threadId:89, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:261)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: DefaultQuartzScheduler_Worker-0 : priority:5, demon:false, threadId:92, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-1 : priority:5, demon:false, threadId:93, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-2 : priority:5, demon:false, threadId:94, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-3 : priority:5, demon:false, threadId:95, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-4 : priority:5, demon:false, threadId:96, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-5 : priority:5, demon:false, threadId:97, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-6 : priority:5, demon:false, threadId:98, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-7 : priority:5, demon:false, threadId:99, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-8 : priority:5, demon:false, threadId:100, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: DefaultQuartzScheduler_Worker-9 : priority:5, demon:false, threadId:101, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
          
           java.lang.Object.wait(Native Method)
           org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
           org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
           org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
          
           Thread: JBossLifeThread : priority:5, demon:false, threadId:112, threadState:WAITING, threadLockName:java.lang.Object@9a4f8e
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:932)
          
           Thread: http-0.0.0.0-8080 : priority:5, demon:true, threadId:113, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
           org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
           org.apache.tomcat.util.net.PoolTcpEndpoint.run(PoolTcpEndpoint.java:647)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: http-0.0.0.0-8080-1 : priority:5, demon:true, threadId:114, threadState:WAITING, threadLockName:org.apache.tomcat.util.net.MasterSlaveWorkerThread@14f8b9a
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
           org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TP-Processor1 : priority:5, demon:true, threadId:115, threadState:WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@344a47
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TP-Processor2 : priority:5, demon:true, threadId:116, threadState:WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@11dc088
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TP-Processor3 : priority:5, demon:true, threadId:117, threadState:WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@13219ed
          
           java.lang.Object.wait(Native Method)
           java.lang.Object.wait(Object.java:485)
           org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TP-Processor4 : priority:5, demon:true, threadId:118, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
           org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
           org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
           org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TP-Monitor : priority:5, demon:true, threadId:119, threadState:TIMED_WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable@8bae7
          
           java.lang.Object.wait(Native Method)
           org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: http-0.0.0.0-8080-2 : priority:5, demon:true, threadId:186, threadState:RUNNABLE, threadLockName:null
          
           sun.management.ThreadImpl.getThreadInfo0(Native Method)
           sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
           sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:123)
           sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
           sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           java.lang.reflect.Method.invoke(Method.java:597)
           org.jboss.system.server.ServerInfo.outputJdk5ThreadMXBeanInfo(ServerInfo.java:657)
           org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:628)
           org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:635)
           org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:635)
           org.jboss.system.server.ServerInfo.listThreadDump(ServerInfo.java:508)
           sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           java.lang.reflect.Method.invoke(Method.java:597)
           org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
           org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
           org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
           org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
           org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
           org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)
           org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)
           org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)
           org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
           org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)
           javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
           javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
           org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
           org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
           org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
           org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
           org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
           org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
           org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
           org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
           org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
           org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
           org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
           org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
           org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
           org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
           org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
           org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: TimeScheduler.Thread : priority:5, demon:true, threadId:512, threadState:TIMED_WAITING, threadLockName:org.jgroups.util.TimeScheduler$TaskQueue@1bc345a
          
           java.lang.Object.wait(Native Method)
           org.jgroups.util.TimeScheduler._run(TimeScheduler.java:391)
           org.jgroups.util.TimeScheduler.access$000(TimeScheduler.java:46)
           org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:135)
           java.lang.Thread.run(Thread.java:619)
          
          
           Thread Group: JBoss Pooled Threads : max priority:10, demon:false
          
          
           Thread Group: System Threads : max priority:10, demon:false
          
           Thread: JBoss System Threads(1)-1 : priority:5, demon:true, threadId:15, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jboss.web.WebServer.run(WebServer.java:324)
           org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-2 : priority:5, demon:true, threadId:19, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jnp.server.Main$AcceptHandler.run(Main.java:441)
           org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-4 : priority:5, demon:true, threadId:86, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:711)
           org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-5 : priority:5, demon:true, threadId:87, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainDatagramSocketImpl.receive0(Native Method)
           java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
           java.net.DatagramSocket.receive(DatagramSocket.java:712)
           org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:615)
           org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-20 : priority:5, demon:true, threadId:765, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-21 : priority:5, demon:true, threadId:767, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-22 : priority:5, demon:true, threadId:770, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-23 : priority:5, demon:true, threadId:773, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-24 : priority:5, demon:true, threadId:775, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: JBoss System Threads(1)-25 : priority:5, demon:true, threadId:791, threadState:TIMED_WAITING, threadLockName:java.lang.Object@141717f
          
           java.lang.Object.wait(Native Method)
           EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
           org.jboss.util.threadpool.MinPooledExecutor.getTask(MinPooledExecutor.java:106)
           EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
           java.lang.Thread.run(Thread.java:619)
          
          
           Thread Group: ConnectionTableGroup : max priority:10, demon:false
          
           Thread: ConnectionTable.AcceptorThread : priority:5, demon:true, threadId:43, threadState:RUNNABLE, threadLockName:null
          
           java.net.PlainSocketImpl.socketAccept(Native Method)
           java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
           java.net.ServerSocket.implAccept(ServerSocket.java:450)
           java.net.ServerSocket.accept(ServerSocket.java:421)
           org.jgroups.blocks.ConnectionTable.run(ConnectionTable.java:349)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: ConnectionTable.Connection.HandlerThread : priority:5, demon:true, threadId:511, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.net.SocketInputStream.read(SocketInputStream.java:182)
           java.io.DataInputStream.readInt(DataInputStream.java:370)
           org.jgroups.blocks.ConnectionTable$Connection.run(ConnectionTable.java:707)
           java.lang.Thread.run(Thread.java:619)
          
          
           Thread Group: QuartzScheduler:DefaultQuartzScheduler : max priority:10, demon:false
          
           Thread: DefaultQuartzScheduler_QuartzSchedulerThread : priority:5, demon:false, threadId:102, threadState:TIMED_WAITING, threadLockName:null
          
           java.lang.Thread.sleep(Native Method)
           org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)
          
          
           Thread Group: RMI Runtime : max priority:10, demon:false
          
           Thread: RMI TCP Connection(608)-68.142.216.41 : priority:5, demon:true, threadId:107, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(593)-68.142.217.46 : priority:5, demon:true, threadId:122, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(idle) : priority:5, demon:true, threadId:125, threadState:TIMED_WAITING, threadLockName:java.util.concurrent.SynchronousQueue$TransferStack@5a1e6b
          
           sun.misc.Unsafe.park(Native Method)
           java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
           java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
           java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
           java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
           java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(613)-68.142.216.101 : priority:5, demon:true, threadId:129, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(580)-68.142.218.103 : priority:5, demon:true, threadId:132, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(idle) : priority:5, demon:true, threadId:140, threadState:TIMED_WAITING, threadLockName:java.util.concurrent.SynchronousQueue$TransferStack@5a1e6b
          
           sun.misc.Unsafe.park(Native Method)
           java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
           java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
           java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
           java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
           java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(610)-68.142.216.41 : priority:5, demon:true, threadId:146, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(567)-68.142.216.45 : priority:5, demon:true, threadId:149, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(612)-68.142.216.169 : priority:5, demon:true, threadId:151, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
           java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
           java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
           java.lang.Thread.run(Thread.java:619)
          
           Thread: RMI TCP Connection(607)-68.142.216.41 : priority:5, demon:true, threadId:153, threadState:RUNNABLE, threadLockName:null
          
           java.net.SocketInputStream.socketRead0(Native Method)
           java.net.SocketInputStream.read(SocketInputStream.java:129)
           java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
           java.io.BufferedInputStream.read(BufferedInputStream.java:237)
           java.io.FilterInputStream.read(FilterInputStream.java:66)
           sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
           sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run


          • 2. Re: EJB3/Socket invoker - connection timeouts

            I'm making progress on this. I tried to reproduce this problem in a test environment, and seem to have found a deadlock state. Here is the stack trace of the SocketServerInvoker thread (the one who calls accept()):

            Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.LRUPool@912
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:439)
             org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
             java.lang.Thread.run(Thread.java:619)
            


            This thread seems to be waiting for a notify() from ServerThread that never comes. I'm still reading code trying to understand how the system gets into this state, but would appreciate some insights from someone more familiar with the code.

            Here is the full thread dump:

            Total Threads: 375
            Total Thread Groups: 9
            
            Thread Group: system : max priority:10, demon:false
            
             Thread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, threadLockName:java.lang.ref.Reference$Lock@d6883
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            
             Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@18caee7
            
             java.lang.Object.wait(Native Method)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
             java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
            
             Thread: Signal Dispatcher : priority:9, demon:true, threadId:4, threadState:RUNNABLE, threadLockName:null
             Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:16, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: RMI Reaper : priority:5, demon:false, threadId:17, threadState:WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@c21a68
            
             java.lang.Object.wait(Native Method)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
             sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: GC Daemon : priority:2, demon:true, threadId:18, threadState:TIMED_WAITING, threadLockName:sun.misc.GC$LatencyLock@16ed0f7
            
             java.lang.Object.wait(Native Method)
             sun.misc.GC$Daemon.run(GC.java:100)
            
             Thread: RMI TCP Accept-4444 : priority:5, demon:true, threadId:21, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: RMI Scheduler(0) : priority:5, demon:true, threadId:27, threadState:TIMED_WAITING, threadLockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ac0a60
            
             sun.misc.Unsafe.park(Native Method)
             java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
             java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1927)
             java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
             java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
             java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
             java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
             java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: RMI TCP Accept-1101 : priority:5, demon:true, threadId:85, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: RMI TCP Accept-4447 : priority:5, demon:true, threadId:88, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
             sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: RMI RenewClean-[68.142.219.229:1101] : priority:5, demon:true, threadId:518, threadState:TIMED_WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@1e70e12
            
             java.lang.Object.wait(Native Method)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
             sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
             java.lang.Thread.run(Thread.java:619)
            
            
             Thread Group: main : max priority:10, demon:false
            
             Thread: DestroyJavaVM : priority:5, demon:false, threadId:9, threadState:RUNNABLE, threadLockName:null
            
             Thread Group: jboss : max priority:10, demon:false
            
             Thread: Timer-0 : priority:5, demon:true, threadId:12, threadState:TIMED_WAITING, threadLockName:java.util.TaskQueue@1af67f0
            
             java.lang.Object.wait(Native Method)
             java.util.TimerThread.mainLoop(Timer.java:509)
             java.util.TimerThread.run(Timer.java:462)
            
             Thread: ScannerThread : priority:5, demon:true, threadId:13, threadState:TIMED_WAITING, threadLockName:null
            
             java.lang.Thread.sleep(Native Method)
             org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)
             org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
            
             Thread: TimeoutFactory-0 : priority:5, demon:true, threadId:20, threadState:WAITING, threadLockName:java.lang.Object@55ff4
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.util.timeout.TimeoutPriorityQueueImpl.poll(TimeoutPriorityQueueImpl.java:253)
             org.jboss.util.timeout.TimeoutPriorityQueueImpl.take(TimeoutPriorityQueueImpl.java:230)
             org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:215)
             org.jboss.util.timeout.TimeoutFactory.access$000(TimeoutFactory.java:41)
             org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:136)
            
             Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:22, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:261)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.LRUPool@912
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:439)
             org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:28, threadState:TIMED_WAITING, threadLockName:null
            
             java.lang.Thread.sleep(Native Method)
             org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:29, threadState:TIMED_WAITING, threadLockName:null
            
             java.lang.Thread.sleep(Native Method)
             org.jboss.ws.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:463)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: SocketServerInvokerThread-68.142.216.101-1 : priority:5, demon:false, threadId:44, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1bd9de3
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-2 : priority:5, demon:false, threadId:45, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@2153fe
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-3 : priority:5, demon:false, threadId:46, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@17c7988
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-5 : priority:5, demon:false, threadId:48, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a2da17
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-6 : priority:5, demon:false, threadId:49, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@174f02c
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-7 : priority:5, demon:false, threadId:50, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@78529d
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-8 : priority:5, demon:false, threadId:51, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@c19fbf
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-9 : priority:5, demon:false, threadId:52, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@63e563
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-10 : priority:5, demon:false, threadId:53, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@82e4f3
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-11 : priority:5, demon:false, threadId:54, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@12f9924
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-12 : priority:5, demon:false, threadId:55, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@14b74a7
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-13 : priority:5, demon:false, threadId:56, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@156d7c8
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-14 : priority:5, demon:false, threadId:57, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1364dcb
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-15 : priority:5, demon:false, threadId:58, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@103d246
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-16 : priority:5, demon:false, threadId:59, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@d0deb5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-17 : priority:5, demon:false, threadId:60, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@136c55
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-18 : priority:5, demon:false, threadId:61, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13af084
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-19 : priority:5, demon:false, threadId:62, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@14596d5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-20 : priority:5, demon:false, threadId:63, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@45ce17
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-21 : priority:5, demon:false, threadId:64, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@15a4523
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-22 : priority:5, demon:false, threadId:65, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13c296b
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-23 : priority:5, demon:false, threadId:66, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@fe2558
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-24 : priority:5, demon:false, threadId:67, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@24c672
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-25 : priority:5, demon:false, threadId:68, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@19bd735
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-26 : priority:5, demon:false, threadId:69, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1fa490e
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-27 : priority:5, demon:false, threadId:70, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@b6a6d6
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-28 : priority:5, demon:false, threadId:71, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@19299f5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-29 : priority:5, demon:false, threadId:72, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@10f34cc
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-30 : priority:5, demon:false, threadId:73, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4e194f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-31 : priority:5, demon:false, threadId:74, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1d592a
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-32 : priority:5, demon:false, threadId:75, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@6b51d8
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-33 : priority:5, demon:false, threadId:76, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a02097
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-34 : priority:5, demon:false, threadId:77, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4ac866
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-35 : priority:5, demon:false, threadId:78, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@3dfcb
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-36 : priority:5, demon:false, threadId:79, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@11b1d80
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.216.101-37 : priority:5, demon:false, threadId:80, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13136e5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: AsynchKeyChangeHandler Thread : priority:5, demon:false, threadId:82, threadState:WAITING, threadLockName:java.lang.Object@17ecdf0
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
             org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: AsynchViewChangeHandler Thread : priority:5, demon:false, threadId:83, threadState:WAITING, threadLockName:java.lang.Object@11cbec9
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
             org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: PooledInvokerAcceptor#0-4446 : priority:5, demon:false, threadId:89, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:261)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: DefaultQuartzScheduler_Worker-0 : priority:5, demon:false, threadId:92, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-1 : priority:5, demon:false, threadId:93, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-2 : priority:5, demon:false, threadId:94, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-3 : priority:5, demon:false, threadId:95, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-4 : priority:5, demon:false, threadId:96, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-5 : priority:5, demon:false, threadId:97, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-6 : priority:5, demon:false, threadId:98, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-7 : priority:5, demon:false, threadId:99, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-8 : priority:5, demon:false, threadId:100, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: DefaultQuartzScheduler_Worker-9 : priority:5, demon:false, threadId:101, threadState:TIMED_WAITING, threadLockName:java.lang.Object@1c94b51
            
             java.lang.Object.wait(Native Method)
             org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
             org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)
             org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
            
             Thread: JBossLifeThread : priority:5, demon:false, threadId:112, threadState:WAITING, threadLockName:java.lang.Object@9a4f8e
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:932)
            
             Thread: http-0.0.0.0-8080 : priority:5, demon:true, threadId:113, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
             org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
             org.apache.tomcat.util.net.PoolTcpEndpoint.run(PoolTcpEndpoint.java:647)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: http-0.0.0.0-8080-1 : priority:5, demon:true, threadId:114, threadState:WAITING, threadLockName:org.apache.tomcat.util.net.MasterSlaveWorkerThread@14f8b9a
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
             org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: TP-Processor1 : priority:5, demon:true, threadId:115, threadState:WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@344a47
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: TP-Processor2 : priority:5, demon:true, threadId:116, threadState:WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$ControlRunnable@11dc088
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: TP-Processor3 : priority:5, demon:true, threadId:117, threadState:RUNNABLE, threadLockName:null
            
             java.net.SocketInputStream.socketRead0(Native Method)
             java.net.SocketInputStream.read(SocketInputStream.java:129)
             java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
             java.io.BufferedInputStream.read(BufferedInputStream.java:317)
             org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
             org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
             org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
             org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
             org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: TP-Processor4 : priority:5, demon:true, threadId:118, threadState:RUNNABLE, threadLockName:null
            
             java.net.PlainSocketImpl.socketAccept(Native Method)
             java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             java.net.ServerSocket.implAccept(ServerSocket.java:450)
             java.net.ServerSocket.accept(ServerSocket.java:421)
             org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
             org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
             org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
             org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: TP-Monitor : priority:5, demon:true, threadId:119, threadState:TIMED_WAITING, threadLockName:org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable@8bae7
            
             java.lang.Object.wait(Native Method)
             org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: http-0.0.0.0-8080-2 : priority:5, demon:true, threadId:186, threadState:WAITING, threadLockName:org.apache.tomcat.util.net.MasterSlaveWorkerThread@e4fc21
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
             org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
             java.lang.Thread.run(Thread.java:619)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4316 : priority:5, demon:false, threadId:1825, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@16c31dc
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: MultiThreadedHttpConnectionManager cleanup : priority:5, demon:true, threadId:1826, threadState:TIMED_WAITING, threadLockName:java.lang.ref.ReferenceQueue$Lock@16e7d5d
            
             java.lang.Object.wait(Native Method)
             java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
             org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4234 : priority:5, demon:false, threadId:1833, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@5730f8
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4324 : priority:5, demon:false, threadId:1835, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@18a04cb
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4282 : priority:5, demon:false, threadId:1836, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@ab627a
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-4166 : priority:5, demon:false, threadId:1846, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@10fbe75
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: Timer-1 : priority:5, demon:true, threadId:1953, threadState:TIMED_WAITING, threadLockName:java.util.TaskQueue@1b9c26e
            
             java.lang.Object.wait(Native Method)
             java.util.TimerThread.mainLoop(Timer.java:509)
             java.util.TimerThread.run(Timer.java:462)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4374 : priority:5, demon:false, threadId:6072, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@13ba863
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4276 : priority:5, demon:false, threadId:6073, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1f24200
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4293 : priority:5, demon:false, threadId:6755, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@9a1ffa
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-3758 : priority:5, demon:false, threadId:6756, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@a0338f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-4355 : priority:5, demon:false, threadId:6757, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1f41688
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4017 : priority:5, demon:false, threadId:6758, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@155cbb9
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4254 : priority:5, demon:false, threadId:6760, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a8a526
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4233 : priority:5, demon:false, threadId:6761, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@bacb4d
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4277 : priority:5, demon:false, threadId:6762, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@fe2145
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-4127 : priority:5, demon:false, threadId:6763, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@15276e1
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4387 : priority:5, demon:false, threadId:6764, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@b4cab5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4345 : priority:5, demon:false, threadId:6765, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@117136e
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-4330 : priority:5, demon:false, threadId:6766, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@142543f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4341 : priority:5, demon:false, threadId:6767, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@11fe652
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4210 : priority:5, demon:false, threadId:6769, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1f2707a
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-3890 : priority:5, demon:false, threadId:6771, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1f3e3a1
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4148 : priority:5, demon:false, threadId:6772, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@10bbc70
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4402 : priority:5, demon:false, threadId:6773, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@168f171
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4026 : priority:5, demon:false, threadId:6774, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@d28e8f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4344 : priority:5, demon:false, threadId:6775, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@139783f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4208 : priority:5, demon:false, threadId:6777, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4cb318
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4394 : priority:5, demon:false, threadId:6778, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@6c40a8
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-4027 : priority:5, demon:false, threadId:6779, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1e5dafc
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-4270 : priority:5, demon:false, threadId:6780, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@378797
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4259 : priority:5, demon:false, threadId:6781, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@403f0e
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4170 : priority:5, demon:false, threadId:6783, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1617d92
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4357 : priority:5, demon:false, threadId:6784, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@299a92
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4367 : priority:5, demon:false, threadId:6785, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1cefabb
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4356 : priority:5, demon:false, threadId:6786, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@c26e9b
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4364 : priority:5, demon:false, threadId:6787, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@15067a4
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4368 : priority:5, demon:false, threadId:6788, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@eafb80
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4379 : priority:5, demon:false, threadId:6824, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@34390a
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4110 : priority:5, demon:false, threadId:6825, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@254d60
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4384 : priority:5, demon:false, threadId:6826, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@12aa87
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4398 : priority:5, demon:false, threadId:6827, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@6d91a2
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-4273 : priority:5, demon:false, threadId:6828, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@67a99d
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4372 : priority:5, demon:false, threadId:6829, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@a3a6b4
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4376 : priority:5, demon:false, threadId:6830, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@fc053
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4308 : priority:5, demon:false, threadId:6831, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a9702b
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4197 : priority:5, demon:false, threadId:6832, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1cb122f
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-4264 : priority:5, demon:false, threadId:6833, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1ad082c
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4301 : priority:5, demon:false, threadId:6834, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1578c45
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4199 : priority:5, demon:false, threadId:6835, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@136d07e
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4098 : priority:5, demon:false, threadId:6836, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@15b4be6
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4400 : priority:5, demon:false, threadId:6837, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1a2d4f3
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4382 : priority:5, demon:false, threadId:6838, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@fc530c
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4360 : priority:5, demon:false, threadId:6839, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@19e7972
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4131 : priority:5, demon:false, threadId:6840, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@569e58
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4361 : priority:5, demon:false, threadId:6841, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@16f1ef3
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-4343 : priority:5, demon:false, threadId:6842, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@f210d4
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-3433 : priority:5, demon:false, threadId:6843, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1571c28
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4365 : priority:5, demon:false, threadId:6844, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4a3208
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.226-4386 : priority:5, demon:false, threadId:6845, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@4a9f3c
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4288 : priority:5, demon:false, threadId:6846, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@12152f1
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.93-4332 : priority:5, demon:false, threadId:6847, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@3bebc1
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.227-4107 : priority:5, demon:false, threadId:6848, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1c48d3d
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.92-3815 : priority:5, demon:false, threadId:6849, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@11a4b66
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4401 : priority:5, demon:false, threadId:6850, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@2dfe8
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.90-4350 : priority:5, demon:false, threadId:6851, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@10d25dc
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4377 : priority:5, demon:false, threadId:6852, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@80d0e5
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4380 : priority:5, demon:false, threadId:6853, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@1fbe6eb
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.91-4407 : priority:5, demon:false, threadId:6854, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@b49c52
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.95-4272 : priority:5, demon:false, threadId:6855, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@756a95
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.225-4371 : priority:5, demon:false, threadId:6856, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@debc76
            
             java.lang.Object.wait(Native Method)
             java.lang.Object.wait(Object.java:485)
             org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)
            
             Thread: SocketServerInvokerThread-68.142.219.94-4399 : priority:5, demon:false, threadId:6857, thr


            • 3. Re: EJB3/Socket invoker - connection timeouts
              gurgstar

              severSocketConstructor.newInstance() reads from the socket, then the socket.setTimeout() is called

              Even if a socket timeout is set, if something just connects to the port and never writes anything, this code will hang indefinitely

              ServerThread.java:183-191

              if(serverSocketConstructor.getParameterTypes().length == 3)
              {
              serverSocketWrapper = (SocketWrapper) serverSocketConstructor.newInstance(new Object[]{socket, metadata, new Integer(timeout)});
              }
              else
              {
              serverSocketWrapper = (SocketWrapper) serverSocketConstructor.newInstance(new Object[]{socket});
              serverSocketWrapper.setTimeout(timeout);
              }

              • 4. Re: EJB3/Socket invoker - connection timeouts
                gurgstar

                Not sure how we got into this state, but its clearly a deadlock...

                (1) SocketServerInvoker is stuck on clientpool.wait()
                (2) maxPoolSize ServerThreads are all stuck on this.wait()
                (3) At least one ServerThread must call clientpool.notify() to wake up the SocketServerInvoker
                (4) SocketServerInvoker must be woken up in order to call notify on at least one ServerThread.

                The deadlock is clear enough, but not sure how we got there other than what appears to be a very small chance of a lost wakeup...

                ServerSocketInvoker:

                 while (thread == null) {
                 synchronized (threadpool) {
                 if (threadpool.size() > 0) {
                 thread = (ServerThread) threadpool.removeFirst();
                 }
                 }
                 if (thread == null) {
                 synchronized (clientpool) {
                 if (clientpool.size() < maxPoolSize) {
                 thread = new ServerThread(socket, this, clientpool, threadpool, getTimeout(), serverSocketClass);
                 newThread = true;
                 }
                 if (thread == null) {
                 clientpool.evict();
                 if (trace) {
                 log.trace("Waiting for a thread...");
                 }
                 clientpool.wait();
                 if (trace) {
                 log.trace("Notified of available thread");
                 }
                 }
                 }
                 }
                 }
                

                ServerThread:
                 synchronized (this) {
                 synchronized (clientpool) {
                 synchronized (threadpool) {
                 if (shutdown) {
                 invoker = null;
                 return; // exit thread
                 } else {
                 clientpool.remove(this);
                 threadpool.add(this);
                 Thread.interrupted(); // clear any interruption so that we can be pooled.
                 clientpool.notify(); //273
                 }
                 }
                 }
                 try {
                 log.debug("begin thread wait");
                 this.wait(); // 280
                 log.debug("WAKEUP in SERVER THREAD");
                 } catch (InterruptedException e) {
                 if (shutdown) {
                 invoker = null;
                 return; // exit thread
                 }
                
                 throw e;
                 }
                 }
                


                There is the potential for lost wakeups in both directions. The ServerSocketInvoker can miss the ServerThread notifies, and the ServerThreads can miss the ServerSocketInvoker's notifies.

                This is how the ServerSocketInvoker could miss the notifies form the ServerThreads (seems very unlikely):

                (1) ServerSocketInvoker sees that the threadpool is empty
                (2) All the ServerThreads then finish processing and call clientpool.notify()
                (3) ServerSocketInvoker calls clientpool.wait() and will wait forever cause it missed the notifies

                This is how the ServerThreads could miss the notifies from the ServerSocketInvoker (seems much more likely):

                (1) A ServerThread finishes processing, calls clientpool.notify() and is then suspended by the scheduler
                (2) ServerSocketInvoker wakes up and calls ServerThread.notify
                (3) The ServerThread then calls this.wait() and waits forever because it missed the ServerSocketInvoker's notify

                Each time this happens, it would take a ServerThread out of comission. Eventually you'd run out of ServerThreads.






                • 5. Re: EJB3/Socket invoker - connection timeouts
                  jahlborn

                  I do believe there is another possible notification issue. I've seen this pattern in the jdk code. I believe a thread which is waiting can get interrupted at about the same time that it is notified. if the InterruptedException is just thrown, that notification will be lost. If you look in some of the jdk code, they have essentially:

                  synchronized(obj) {
                   try {
                   obj.wait();
                   } catch(InterruptedException e) {
                   obj.notify();
                   throw e;
                   }
                  }
                  


                  This way an interrupted thread will not cause a notification to be lost. obviously, you can generate spurious notifications, so your code needs to handle that as well. but generally, extra notifications is better than lost notifications.

                  To bring it back to the referenced code:
                  (1) SocketServerInvoker could be doing the clientpool.wait()
                  (2) ServerThread sends a notification
                  (3) SocketServerInvoker gets interrupted and notified and bails out, swallowing the notification

                  I'm not sure what scenarios might cause the SocketServerInvoker to be interrupted, but certainly system stress might do it.


                  • 6. Re: EJB3/Socket invoker - connection timeouts
                    ron_sigal

                     


                    This is how the ServerThreads could miss the notifies from the ServerSocketInvoker (seems much more likely):

                    (1) A ServerThread finishes processing, calls clientpool.notify() and is then suspended by the scheduler
                    (2) ServerSocketInvoker wakes up and calls ServerThread.notify
                    (3) The ServerThread then calls this.wait() and waits forever because it missed the ServerSocketInvoker's notify

                    Each time this happens, it would take a ServerThread out of comission. Eventually you'd run out of ServerThreads.


                    Before we get in too deep here, check out thread "Important bug in thread pool management" (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=80814 )
                    and JIRA issue JBREM-409 (http://jira.jboss.com/jira/browse/JBREM-409). This bug was fixed as of Remoting version 1.4.5.GA, but JBoss Application Server 4.0.5, for example, comes with Remoting 1.4.3.GA. You should be able to just drop in jboss-remoting.jar from, say, version 1.4.6.GA (the latest in the 1.x line). Moreover, I believe that the latest version of EJB3 should run with the newest version of Remoting, which is 2.2.0.SP4.

                    Let me know whether upgrading solves the problem.

                    -Ron

                    • 7. Re: EJB3/Socket invoker - connection timeouts

                      Thanks Ron. Yeah, I came across JBREM-409 a few days ago, and had already applied that patch from 1.4.6 (which seems to have been tagged in CVS but never released) to our code. So this problem definitely seems to still exist in 1.4.6.

                      Upgrading the EJB implementation and going to Remoting 2.2.x is a bit more drastic, as we've been testing and tuning the current code base for months, and an upgrade of that magnitude could invalidate a lot of work.

                      • 8. Re: EJB3/Socket invoker - connection timeouts
                        ron_sigal

                         


                        It appears that this thread, which ought to be quickly finding a ServerThread to handle the incoming request, calling ServerThread.start(), and getting back to the call to serverSocket.accept() as quickly as possible, is instead doing some socket I/O. ServerSocketWrapper.java has a method called createInputStream() that creates a new ObjectInputStream. The stack trace shows that the JBossObjectInputStream constructor is reading some bytes from the input stream, but I don't think that this problem is specific to JBoss Serialization, as java.io.ObjectInputStream's constructor also reads bytes from the stream.

                        It seems to me that this could be the cause of the problem that we are seeing. Any time that you're doing a blocking socket read like this, you can potentially just sit for a while until the socket times out. When you've got a thread calling serverSocket.accept(), you can't normally afford to do something like this on that thread.

                        Does this analysis make sense? Any thoughts/suggestions would be greatly appreciated.


                        It does indeed make sense. See JIRA issue JBREM-666 (http://jira.jboss.com/jira/browse/JBREM-666 ).

                        This problem was fixed in the Remoting 2.2.0 releases, the most recent of which is 2.2.0.SP4. Which version of the Application Server are you using? The latest GA release, 4.2.0.GA, comes with Remoting 2.2.0.SP4. If you can't upgrade that far, then you might try getting the latest version of EJB3, which, I believe, should work with Remoting 2.2.0. This could cause problems for other Remoting clients which depend on an older version of Remoting. In that case, look into scoped deployment, which allows different applications to use different library versions. See, for example, the wiki page "ClassLoadingConfiguration" (http://wiki.jboss.org/wiki/Wiki.jsp?page=ClassLoadingConfiguration).

                        -Ron

                        • 9. Re: EJB3/Socket invoker - connection timeouts
                          ron_sigal

                           

                          "javajedi" wrote:
                          Thanks Ron. Yeah, I came across JBREM-409 a few days ago, and had already applied that patch from 1.4.6 (which seems to have been tagged in CVS but never released) to our code. So this problem definitely seems to still exist in 1.4.6.


                          The 1.4.6.GA jar is at http://repository.jboss.com/jboss/remoting/1.4.6.GA/lib. You might try it, just in case. Note that it looks like the fix for JBREM-409 involved some changes to SocketServerInvoker as well as ServerThread (though I don't remember what they were, off the top of my head).

                          -Ron

                          • 10. Re: EJB3/Socket invoker - connection timeouts

                            The only relevant change that I saw to SocketServerInvoker (that I already patched) was the call to
                            serverSocket.setReuseAddress(true);

                            Was there more that I'm missing?

                            • 11. Re: EJB3/Socket invoker - connection timeouts

                              BTW, we are using JBoss 4.0.4 and ejb 3.0-rc9.

                              • 12. Re: EJB3/Socket invoker - connection timeouts

                                A few months ago, I think I tried using JBoss Remoting 2.x with EJB 3.0-rc9, and it didn't work. Should it?

                                • 13. Re: EJB3/Socket invoker - connection timeouts
                                  ron_sigal

                                   

                                  "javajedi" wrote:
                                  The only relevant change that I saw to SocketServerInvoker (that I already patched) was the call to
                                  serverSocket.setReuseAddress(true);

                                  Was there more that I'm missing?


                                  I guess you're right. I can't see anything else.

                                  • 14. Re: EJB3/Socket invoker - connection timeouts
                                    ron_sigal

                                     

                                    "javajedi" wrote:
                                    A few months ago, I think I tried using JBoss Remoting 2.x with EJB 3.0-rc9, and it didn't work. Should it?


                                    I can tell you that the Application Server 4.2.0.GA comes with Remoting 2.2.0.SP4, so the version of EJB3 in that release must work with that version of Remoting. I'm not sure which version of EJB3 that is, though.


                                    I'm going to refer you to the EJB3 forum (http://www.jboss.com/index.html?module=bb&op=viewforum&f=221) .

                                    1 2 Previous Next