1 2 3 Previous Next 30 Replies Latest reply on Aug 28, 2008 9:42 AM by siimk123

    BisocketClientInvoker deadlock

    ykonrad

      Hi,

      We are facing a deadlock in the JMS server due to the following stack trace:

      This is the same thread from this forum, with no solution, we are not working behind a firewall: http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4140268#4140268

      Thread: Notify Thread : priority:5, demon:false, threadId:83, threadState:TIMED_WAITING, lockName:java.util.HashSet@7d9dbajava.lang.Object.wait(Native Method)org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)org.jboss.remoting.Client.invoke(Client.java:1634)org.jboss.remoting.Client.invoke(Client.java:548)org.jboss.remoting.Client.invokeOneway(Client.java:598)org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)org.jboss.remoting.Client.invoke(Client.java:1634)org.jboss.remoting.Client.invoke(Client.java:548)org.jboss.remoting.Client.invoke(Client.java:536)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:324)com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)
      


      Thanks,

      Yuval.

        • 1. Re: BisocketClientInvoker deadlock
          ykonrad

          Sorry,
          My mistake posting the stack trace. Again:

          Thread: Notify Thread : priority:5, demon:false, threadId:83, threadState:TIMED_WAITING, lockName:java.util.HashSet@7d9dba
          java.lang.Object.wait(Native Method)
          org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
          org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
          org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
          org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
          org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
          org.jboss.remoting.Client.invoke(Client.java:1634)
          org.jboss.remoting.Client.invoke(Client.java:548)
          org.jboss.remoting.Client.invokeOneway(Client.java:598)
          org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
          org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
          org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
          org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
          org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
          org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
          org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
          org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
          org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
          org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
          org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
          org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
          org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
          org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
          org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
          org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
          org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
          org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
          sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          java.lang.reflect.Method.invoke(Method.java:585)
          org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
          org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
          org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
          org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
          org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
          org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
          org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
          org.jboss.remoting.Client.invoke(Client.java:1634)
          org.jboss.remoting.Client.invoke(Client.java:548)
          org.jboss.remoting.Client.invoke(Client.java:536)
          org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
          org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
          org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
          org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
          org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
          org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
          org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
          org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
          org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
          org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
          org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
          org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
          org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
          org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
          org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
          org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
          org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
          org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
          org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
          org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
          org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
          org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
          org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
          org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
          org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
          com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:324)
          com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)
          com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)
          com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)
          com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)
          com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)
          com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)
          
          
          


          • 2. Re: BisocketClientInvoker deadlock
            timfox

            Please state what versions of JBM and JBR you are using when making a post. Without that information no one will be able to help you.

            Please also ensure you have the correct versions of JBM and JBR on both the client and server side (as explained in the user guide).

            Also please paste a java stack dump showing the deadlock.

            Thanks.

            • 3. Re: BisocketClientInvoker deadlock
              ykonrad

              My mistake(again)

              JBM 1.4.0.SP3
              Remoting 2.2.2.SP7

              Regarding the full stack trace - I am not able to get it right now, but this specific single Thread dump is the root cause.

              • 4. Re: BisocketClientInvoker deadlock
                timfox

                Next time it occurs, please get a kill -3 thread dump, and we'll take it from there.

                Thanks.

                • 5. Re: BisocketClientInvoker deadlock
                  ykonrad



                  Ok, I got it from the jmx-console

                  
                  JBoss
                  JMX MBean Operation Result listThreadDump()
                  Back to Agent View Back to MBean View Reinvoke MBean Operation
                  
                  Total Threads: 232
                  Total Thread Groups: 10
                  
                  Thread Group: system : max priority:10, demon:false
                  
                   Thread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, lockName:java.lang.ref.Reference$Lock@183c5e
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
                  
                   Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@18d6b41
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
                   java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
                   java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
                  
                   Thread: Signal Dispatcher : priority:9, demon:true, threadId:4, threadState:RUNNABLE, lockName:null
                   Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:17, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: RMI Reaper : priority:5, demon:false, threadId:19, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@1b7ceb
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
                   java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
                   sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: GC Daemon : priority:2, demon:true, threadId:20, threadState:TIMED_WAITING, lockName:sun.misc.GC$LatencyLock@a667f4
                  
                   java.lang.Object.wait(Native Method)
                   sun.misc.GC$Daemon.run(GC.java:100)
                  
                   Thread: RMI LeaseChecker : priority:5, demon:true, threadId:23, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: RMI TCP Accept-4449 : priority:5, demon:true, threadId:30, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
                   java.lang.Thread.run(Thread.java:595)
                  
                  
                   Thread Group: main : max priority:10, demon:false
                  
                   Thread: DestroyJavaVM : priority:5, demon:false, threadId:10, threadState:RUNNABLE, lockName:null
                  
                   Thread Group: jboss : max priority:10, demon:false
                  
                   Thread: Timer-0 : priority:5, demon:true, threadId:13, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@15b3b26
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: ScannerThread : priority:5, demon:true, threadId:14, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)
                   org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
                  
                   Thread: Timer-1 : priority:5, demon:true, threadId:18, threadState:WAITING, lockName:java.util.TaskQueue@e96db5
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   java.util.TimerThread.mainLoop(Timer.java:483)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Listener:44208 : priority:5, demon:true, threadId:24, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)
                  
                   Thread: Thread-4 : priority:5, demon:true, threadId:26, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)
                  
                   Thread: Listener:44211 : priority:5, demon:true, threadId:29, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)
                  
                   Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:31, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: AcceptorThread#0:4446 : priority:5, demon:false, threadId:32, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: ServerSocketRefresh : priority:5, demon:true, threadId:33, threadState:WAITING, lockName:java.lang.Object@1e9ce35
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                  
                   Thread: AcceptorThread#0:3874 : priority:5, demon:false, threadId:34, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
                   org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: ServerSocketRefresh : priority:5, demon:true, threadId:35, threadState:WAITING, lockName:java.lang.Object@3f82bd
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                  
                   Thread: AcceptorThread#0:4457 : priority:5, demon:false, threadId:36, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: ServerSocketRefresh : priority:5, demon:true, threadId:37, threadState:WAITING, lockName:java.lang.Object@1a04c93
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                  
                   Thread: secondaryServerSocketThread : priority:5, demon:true, threadId:38, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:871)
                  
                   Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:39, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: WorkerThread#0[64.103.124.247:18705] : priority:5, demon:false, threadId:40, threadState:BLOCKED, lockName:java.util.Collections$SynchronizedRandomAccessList@1035617
                  
                   java.util.Collections$SynchronizedList.add(Collections.java:1826)
                   org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                   org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                   org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                   org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                   org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                   org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                   org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                  
                   Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:41, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1549)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Thread-8 : priority:5, demon:true, threadId:42, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.jboss.console.plugins.AOPLister$1.run(AOPLister.java:882)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: DefaultQuartzScheduler_Worker-1 : priority:5, demon:false, threadId:43, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@90399
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-2 : priority:5, demon:false, threadId:44, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@1cb6938
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-3 : priority:5, demon:false, threadId:45, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@168fe4d
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-4 : priority:5, demon:false, threadId:46, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@fa22a
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-5 : priority:5, demon:false, threadId:47, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@5fa7bf
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-6 : priority:5, demon:false, threadId:48, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@1fdf2f1
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-7 : priority:5, demon:false, threadId:49, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@158db65
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-8 : priority:5, demon:false, threadId:50, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@ac042d
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-9 : priority:5, demon:false, threadId:51, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@14651c8
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: DefaultQuartzScheduler_Worker-10 : priority:5, demon:false, threadId:52, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@1a35777
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Timer-2 : priority:5, demon:true, threadId:54, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@c3498b
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: JCA PoolFiller : priority:5, demon:false, threadId:55, threadState:WAITING, lockName:java.util.LinkedList@1118090
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:87)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Thread-10 : priority:5, demon:true, threadId:56, threadState:TIMED_WAITING, lockName:com.arjuna.ats.arjuna.coordinator.TransactionReaper@f8f4f
                  
                   java.lang.Object.wait(Native Method)
                   com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)
                  
                   Thread: Thread-11 : priority:5, demon:true, threadId:57, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Thread-12 : priority:5, demon:false, threadId:58, threadState:WAITING, lockName:java.lang.Object@189bedb
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                   EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: XML-RPC Weblistener : priority:5, demon:false, threadId:59, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.apache.xmlrpc.WebServer.run(Unknown Source)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Timer-3 : priority:5, demon:true, threadId:60, threadState:WAITING, lockName:EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock@513960
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingInMemory(MessagingPostOffice.java:2280)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalRemoveBinding(MessagingPostOffice.java:1902)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBinding(MessagingPostOffice.java:464)
                   org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:400)
                   org.jboss.jms.server.connectionmanager.SimpleConnectionManager.closeConsumersForClientVMID(SimpleConnectionManager.java:377)
                   org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(SimpleConnectionManager.java:173)
                   org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(SimpleConnectionManager.java:200)
                   org.jboss.remoting.ConnectionNotifier.connectionLost(ConnectionNotifier.java:55)
                   org.jboss.remoting.Lease.notifyClientLost(Lease.java:211)
                   org.jboss.remoting.Lease.access$500(Lease.java:39)
                   org.jboss.remoting.Lease$LeaseTimerTask.run(Lease.java:249)
                   java.util.TimerThread.mainLoop(Timer.java:512)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Timer-4 : priority:5, demon:false, threadId:61, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@8dced0
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: EJB-Timer-4[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmSyncAuthEJB] : priority:5, demon:false, threadId:65, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@4cb69
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: EJB-Timer-5[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmNotificationEJB] : priority:5, demon:false, threadId:66, threadState:BLOCKED, lockName:org.jboss.jms.server.ServerPeer@c660a9
                  
                   org.jboss.jms.server.ServerPeer.getDefaultDLQInstance(ServerPeer.java:1158)
                   org.jboss.jms.server.endpoint.ServerSessionEndpoint.(ServerSessionEndpoint.java:218)
                   org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:246)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$createSessionDelegate$aop(ConnectionAdvised.java:75)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised.createSessionDelegate(ConnectionAdvised.java)
                   org.jboss.jms.wireformat.ConnectionCreateSessionDelegateRequest.serverInvoke(ConnectionCreateSessionDelegateRequest.java:87)
                   org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                   org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                   org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                   org.jboss.remoting.Client.invoke(Client.java:1634)
                   org.jboss.remoting.Client.invoke(Client.java:548)
                   org.jboss.remoting.Client.invoke(Client.java:536)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$createSessionDelegate$aop(ClientConnectionDelegate.java:186)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.StateCreationAspect.handleCreateSessionDelegate(StateCreationAspect.java:128)
                   org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect2.invoke(StateCreationAspect2.java)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.ConnectionAspect.handleCreateSessionDelegate(ConnectionAspect.java:163)
                   sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate.createSessionDelegate(ClientConnectionDelegate.java)
                   org.jboss.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:269)
                   org.jboss.jms.client.JBossConnection.createSession(JBossConnection.java:91)
                   com.cisco.nm.dmm.ejb.NotificationTimerBean.publishMessage(NotificationTimerBean.java:233)
                   com.cisco.nm.dmm.ejb.NotificationTimerBean.timeoutHandler(NotificationTimerBean.java:188)
                   sun.reflect.GeneratedMethodAccessor430.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                   org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                   org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                   org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                   org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.stateless.StatelessContainer.callTimeout(StatelessContainer.java:151)
                   org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
                   java.util.TimerThread.mainLoop(Timer.java:512)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: EJB-Timer-6[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmQueueNotificationEJB] : priority:5, demon:false, threadId:67, threadState:WAITING, lockName:EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock@1f3fed3
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.getBindingForQueueName(MessagingPostOffice.java:548)
                   org.jboss.jms.server.ServerPeer.getDefaultDLQInstance(ServerPeer.java:1169)
                   org.jboss.jms.server.endpoint.ServerSessionEndpoint.(ServerSessionEndpoint.java:218)
                   org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:246)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$createSessionDelegate$aop(ConnectionAdvised.java:75)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.server.endpoint.advised.ConnectionAdvised.createSessionDelegate(ConnectionAdvised.java)
                   org.jboss.jms.wireformat.ConnectionCreateSessionDelegateRequest.serverInvoke(ConnectionCreateSessionDelegateRequest.java:87)
                   org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                   org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                   org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                   org.jboss.remoting.Client.invoke(Client.java:1634)
                   org.jboss.remoting.Client.invoke(Client.java:548)
                   org.jboss.remoting.Client.invoke(Client.java:536)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$createSessionDelegate$aop(ClientConnectionDelegate.java:186)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.StateCreationAspect.handleCreateSessionDelegate(StateCreationAspect.java:128)
                   org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect2.invoke(StateCreationAspect2.java)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.ConnectionAspect.handleCreateSessionDelegate(ConnectionAspect.java:163)
                   sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
                   org.jboss.jms.client.delegate.ClientConnectionDelegate.createSessionDelegate(ClientConnectionDelegate.java)
                   org.jboss.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:269)
                   org.jboss.jms.client.JBossConnection.createSession(JBossConnection.java:91)
                   com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsSender.publishMessageToDMMQueues(DMMJ2EEQueueNtfsSender.java:102)
                   com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsSender.initiateNotificationTask(DMMJ2EEQueueNtfsSender.java:72)
                   com.cisco.nm.dmm.common.services.QueueDispatcher.updateCacheAndPublishToWorker(QueueDispatcher.java:263)
                   com.cisco.nm.dmm.ejb.QueueNotificationTimerBean.timeoutHandler(QueueNotificationTimerBean.java:196)
                   sun.reflect.GeneratedMethodAccessor432.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                   org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                   org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                   org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                   org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                   org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                   org.jboss.ejb3.stateless.StatelessContainer.callTimeout(StatelessContainer.java:151)
                   org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
                   java.util.TimerThread.mainLoop(Timer.java:512)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Dummy Log processing thread : priority:5, demon:true, threadId:71, threadState:TIMED_WAITING, lockName:java.lang.Object@1decfa5
                  
                   java.lang.Object.wait(Native Method)
                   com.cisco.nm.cmp.system.os.logger.DummyLogger$1.run(DummyLogger.java:163)
                  
                   Thread: IdleRemover : priority:5, demon:true, threadId:77, threadState:TIMED_WAITING, lockName:java.util.ArrayList@1127751
                  
                   java.lang.Object.wait(Native Method)
                   org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Send Thread : priority:5, demon:false, threadId:80, threadState:TIMED_WAITING, lockName:com.cisco.nm.cmp.gw.adapters.ptp.util.BlockingQueue@1579172
                  
                   java.lang.Object.wait(Native Method)
                   com.cisco.nm.cmp.gw.adapters.ptp.util.BlockingQueue.dequeue(BlockingQueue.java:57)
                   com.cisco.nm.cmp.gw.ptpclient.protocol.SendThread.run(SendThread.java:54)
                  
                   Thread: Recieve Thread : priority:5, demon:false, threadId:81, threadState:RUNNABLE, lockName:null
                  
                   java.net.SocketInputStream.socketRead0(Native Method)
                   java.net.SocketInputStream.read(SocketInputStream.java:129)
                   java.net.SocketInputStream.read(SocketInputStream.java:182)
                   java.io.DataInputStream.readInt(DataInputStream.java:353)
                   com.cisco.nm.cmp.gw.adapters.ptp.protocol.SocketConnection.receive(SocketConnection.java:93)
                   com.cisco.nm.cmp.gw.adapters.ptp.protocol.SocketConnection.receive(SocketConnection.java:85)
                   com.cisco.nm.cmp.gw.ptpclient.protocol.ReceiveThread.run(ReceiveThread.java:71)
                  
                   Thread: Connection State : priority:5, demon:false, threadId:82, threadState:WAITING, lockName:com.cisco.nm.cmp.gw.adapters.ptp.protocol.ConnectionState@e054b
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   com.cisco.nm.cmp.gw.ptpclient.protocol.ConnectionStateThread.run(ConnectionStateThread.java:40)
                  
                   Thread: Notify Thread : priority:5, demon:false, threadId:83, threadState:TIMED_WAITING, lockName:java.util.HashSet@7d9dba
                  
                   java.lang.Object.wait(Native Method)
                   org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
                   org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
                   org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
                   org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
                   org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                   org.jboss.remoting.Client.invoke(Client.java:1634)
                   org.jboss.remoting.Client.invoke(Client.java:548)
                   org.jboss.remoting.Client.invokeOneway(Client.java:598)
                   org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                   org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                   org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
                   org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
                   org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                   org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                   org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                   org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                   org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                   org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                   org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                   org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                   org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                   org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                   org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                   org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                   org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                   sun.reflect.GeneratedMethodAccessor438.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                   org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                   org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                   org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                   org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                   org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                   org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                   org.jboss.remoting.Client.invoke(Client.java:1634)
                   org.jboss.remoting.Client.invoke(Client.java:548)
                   org.jboss.remoting.Client.invoke(Client.java:536)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                   org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                   org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
                   org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                   org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                   org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                   org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                   org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                   org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
                   org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                   org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                   org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                   org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                   org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                   org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                   org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                   org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                   org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                   com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:324)
                   com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)
                   com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)
                   com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)
                   com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)
                   com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)
                   com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)
                  
                   Thread: UserConnectionPollerThread : priority:5, demon:false, threadId:84, threadState:BLOCKED, lockName:com.cisco.nm.cmp.gjc.cm.UserConnectionImpl@fe3d5c
                  
                   com.cisco.nm.cmp.gjc.cm.UserConnectionManager.expireConnections(UserConnectionManager.java:278)
                   com.cisco.nm.cmp.gjc.cm.UserConnectionPoller.run(UserConnectionPoller.java:62)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Logger notifications thread : priority:5, demon:true, threadId:97, threadState:TIMED_WAITING, lockName:java.lang.Object@16e91af
                  
                   java.lang.Object.wait(Native Method)
                   com.cisco.nm.cmp.system.os.logger.AbstractLoggerImpl$1.run(AbstractLoggerImpl.java:333)
                  
                   Thread: WorkerThread#1[64.103.124.247:18850] : priority:5, demon:false, threadId:101, threadState:WAITING, lockName:org.jboss.remoting.transport.socket.ServerThread@ede434
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:212)
                  
                   Thread: Scheduler1_Worker-1 : priority:5, demon:false, threadId:102, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@1eae3c
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Scheduler1_Worker-2 : priority:5, demon:false, threadId:103, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@41135d
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Scheduler1_Worker-3 : priority:5, demon:false, threadId:104, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@5e490e
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Scheduler1_Worker-4 : priority:5, demon:false, threadId:105, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@e5518
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Scheduler1_Worker-5 : priority:5, demon:false, threadId:106, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@f4c4be
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: Scheduler1_Worker-6 : priority:5, demon:false, threadId:107, threadState:TIMED_WAITING, lockName:org.quartz.simpl.SimpleThreadPool$WorkerThread@1a0abfa
                  
                   java.lang.Object.wait(Native Method)
                   org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                  
                   Thread: QuartzScheduler_Scheduler1-1_MisfireHandler : priority:5, demon:false, threadId:109, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3829)
                  
                   Thread: Timer-6 : priority:5, demon:false, threadId:114, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@17ac076
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Timer-7 : priority:5, demon:true, threadId:121, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1f1a48f
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Thread-40 : priority:5, demon:false, threadId:123, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   com.cisco.nm.cmp.nbi.wsdm.platform.background.RuntimeSystemPropertiesWatcher$ExternalSystemParamtersMonitorThread.run(RuntimeSystemPropertiesWatcher.java:81)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Timer-8 : priority:5, demon:true, threadId:124, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@16ada85
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: Thread-43 : priority:5, demon:false, threadId:126, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   com.cisco.nm.cmp.nbi.wsdm.platform.background.BackgroundFileMonitorProcess$FileMonitorThread.run(BackgroundFileMonitorProcess.java:92)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: Timer-9 : priority:5, demon:true, threadId:127, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@30724a
                  
                   java.lang.Object.wait(Native Method)
                   java.util.TimerThread.mainLoop(Timer.java:509)
                   java.util.TimerThread.run(Timer.java:462)
                  
                   Thread: JBossLifeThread : priority:5, demon:false, threadId:130, threadState:WAITING, lockName:java.lang.Object@9f585e
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)
                  
                   Thread: Thread-5 : priority:5, demon:false, threadId:27, threadState:TIMED_WAITING, lockName:null
                  
                   java.lang.Thread.sleep(Native Method)
                   com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:248)
                   com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)
                  
                   Thread: http-0.0.0.0-8080-Acceptor-0 : priority:5, demon:true, threadId:131, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
                   org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-1 : priority:5, demon:true, threadId:132, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@162bb08
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: ajp-0.0.0.0-8009-Acceptor-0 : priority:5, demon:true, threadId:133, threadState:RUNNABLE, lockName:null
                  
                   java.net.PlainSocketImpl.socketAccept(Native Method)
                   java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                   java.net.ServerSocket.implAccept(ServerSocket.java:450)
                   java.net.ServerSocket.accept(ServerSocket.java:421)
                   org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
                   org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-2 : priority:5, demon:true, threadId:134, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@e533f6
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-3 : priority:5, demon:true, threadId:135, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1b67073
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-4 : priority:5, demon:true, threadId:136, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@90bf1c
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-5 : priority:5, demon:true, threadId:137, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@f2e968
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-6 : priority:5, demon:true, threadId:138, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1b9c1e5
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-7 : priority:5, demon:true, threadId:139, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@68e2e5
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-8 : priority:5, demon:true, threadId:140, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@86832
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-9 : priority:5, demon:true, threadId:141, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1f0d096
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-10 : priority:5, demon:true, threadId:142, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@10e603d
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-11 : priority:5, demon:true, threadId:143, threadState:RUNNABLE, lockName:null
                  
                   sun.management.ThreadImpl.getThreadInfo0(Native Method)
                   sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)
                   sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
                   sun.reflect.GeneratedMethodAccessor618.invoke(Unknown Source)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.system.server.ServerInfo.outputJdk5ThreadMXBeanInfo(ServerInfo.java:662)
                   org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:631)
                   org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)
                   org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)
                   org.jboss.system.server.ServerInfo.listThreadDump(ServerInfo.java:508)
                   sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                   sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   java.lang.reflect.Method.invoke(Method.java:585)
                   org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                   org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                   org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
                   org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                   org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                   org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)
                   org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)
                   org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)
                   org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
                   org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)
                   javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
                   javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
                   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
                   org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                   org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
                   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                   org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                   org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
                   org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
                   org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
                   org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
                   org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
                   org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
                   org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
                   org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                   org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
                   org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
                   org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-12 : priority:5, demon:true, threadId:144, threadState:RUNNABLE, lockName:null
                  
                   java.net.SocketInputStream.socketRead0(Native Method)
                   java.net.SocketInputStream.read(SocketInputStream.java:129)
                   org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)
                   org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
                   org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)
                   org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-13 : priority:5, demon:true, threadId:145, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@d07485
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-14 : priority:5, demon:true, threadId:146, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@191f657
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                   org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                   java.lang.Thread.run(Thread.java:595)
                  
                   Thread: http-0.0.0.0-8080-15 : priority:5, demon:true, threadId:147, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1c3e019
                  
                   java.lang.Object.wait(Native Method)
                   java.lang.Object.wait(Object.java:474)
                  


                  • 6. Re: BisocketClientInvoker deadlock
                    timfox

                    Can you get the stack using kill -3 please (or CTRL-BREAK on windows)?

                    • 7. Re: BisocketClientInvoker deadlock
                      ykonrad


                      Sorry, I can't. I took this stack from the browser history and the server was restarted already.
                      I am not sure I will be able to repreduce it in the short term.

                      • 8. Re: BisocketClientInvoker deadlock
                        ykonrad

                        I managed to get it again. Here is the full stack trace from kill -3 command:

                        Full thread dump Java HotSpot(TM) Server VM (1.5.0_08-b03 mixed mode):
                        
                        "RMI TCP Connection(353)-171.69.71.29" daemon prio=10 tid=0x03ab76e8 nid=0x3884 runnable [0xd3dff000..0xd3dffc70]
                         at java.net.SocketInputStream.socketRead0(Native Method)
                         at java.net.SocketInputStream.read(SocketInputStream.java:129)
                         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                         at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                         - locked <0xaa9a90a0> (a java.io.BufferedInputStream)
                         at java.io.FilterInputStream.read(FilterInputStream.java:66)
                         at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
                         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3520" daemon prio=10 tid=0x027dc648 nid=0x3867 waiting for monitor entry [0xd349c000..0xd349f8f0]
                         at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:117)
                         - waiting to lock <0x473f6e30> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                         at com.cisco.nm.cmp.gjc.GWConnectionImpl.execute(GWConnectionImpl.java:225)
                         at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:820)
                         at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                         at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                         at sun.reflect.GeneratedMethodAccessor474.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                         at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                         at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                         at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                         at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:79)
                         at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:70)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:103)
                         at $Proxy133.execute(Unknown Source)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigArchiveUtility.getUnitMachineforNE(ConfigArchiveUtility.java:288)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigArchiveUtility.getCompleteFilePath(ConfigArchiveUtility.java:428)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.executeConfigFetchCommand(ConfigCollector.java:216)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.getConfiguration(ConfigCollector.java:150)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigBackupService.onConfigChangeEvent(ConfigBackupService.java:410)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ArchiveConfigBean.onConfigChangeEvent(ArchiveConfigBean.java:53)
                         at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                         at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                         at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
                         at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
                         at $Proxy301.onConfigChangeEvent(Unknown Source)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.sendEvent(ConfigChangeListenerMDB.java:160)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.onMessage(ConfigChangeListenerMDB.java:84)
                         at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPolicy.invokeInNoTx(TxPolicy.java:66)
                         at org.jboss.aspects.tx.TxInterceptor$NotSupported.invoke(TxInterceptor.java:112)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy294.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3519" daemon prio=10 tid=0x03a03920 nid=0x385f in Object.wait() [0xd722b000..0xd722f8f0]
                         at java.lang.Object.wait(Native Method)
                         - waiting on <0xcd173de8> (a java.util.HashSet)
                         at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
                         - locked <0xcd173de8> (a java.util.HashSet)
                         at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
                         at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
                         at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
                         at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                         at org.jboss.remoting.Client.invoke(Client.java:1634)
                         at org.jboss.remoting.Client.invoke(Client.java:548)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                         at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                         at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
                         - locked <0x6ce4bee0> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
                         at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                         - locked <0x6ce4bf80> (a java.lang.Object)
                         at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                         at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                         at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                         at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                         at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                         at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                         - locked <0x6ce4c040> (a java.lang.Object)
                         at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                         at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                         at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                         at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                         at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                         at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                         at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                         at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                         at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                         at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                         at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                         at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                         at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                         at org.jboss.remoting.Client.invoke(Client.java:1634)
                         at org.jboss.remoting.Client.invoke(Client.java:548)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:786)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:776)
                         at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
                         at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                         at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                         at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134)
                         at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141)
                         at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117)
                         at sun.reflect.GeneratedMethodAccessor632.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy168.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10417" daemon prio=10 tid=0x03a03758 nid=0x385e in Object.wait() [0xd1fff000..0xd1fff970]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2fe628> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3518" daemon prio=10 tid=0x011696f0 nid=0x385d in Object.wait() [0xd2bdc000..0xd2bdfbf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                         - locked <0x46c2fb58> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                         at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                         at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                         at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                         at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                         at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                         at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                         at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                         at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                         at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                         at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                         at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                         at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                         at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                         at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                         at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                         at org.jboss.remoting.Client.invoke(Client.java:1634)
                         at org.jboss.remoting.Client.invoke(Client.java:548)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:786)
                         at org.jboss.remoting.Client.invokeOneway(Client.java:776)
                         at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
                         at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                         at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                         at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                         at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                         at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134)
                         at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141)
                         at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117)
                         at sun.reflect.GeneratedMethodAccessor632.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy168.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3517" daemon prio=10 tid=0x02047b68 nid=0x385c waiting on condition [0xd1f0d000..0xd1f0fc70]
                         at java.lang.Thread.sleep(Native Method)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.executeConfigFetchCommand(ConfigCollector.java:240)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.getConfiguration(ConfigCollector.java:150)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigBackupService.onConfigChangeEvent(ConfigBackupService.java:410)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ArchiveConfigBean.onConfigChangeEvent(ArchiveConfigBean.java:53)
                         at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                         at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                         at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
                         at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
                         at $Proxy301.onConfigChangeEvent(Unknown Source)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.sendEvent(ConfigChangeListenerMDB.java:160)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.onMessage(ConfigChangeListenerMDB.java:84)
                         at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPolicy.invokeInNoTx(TxPolicy.java:66)
                         at org.jboss.aspects.tx.TxInterceptor$NotSupported.invoke(TxInterceptor.java:112)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy294.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10416" daemon prio=10 tid=0x024b7048 nid=0x385b in Object.wait() [0xd286f000..0xd286faf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd32db98> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10413" daemon prio=10 tid=0x025ca9f8 nid=0x3858 in Object.wait() [0xd38ff000..0xd38ffa70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd333660> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10411" daemon prio=10 tid=0x0326fd78 nid=0x3856 in Object.wait() [0xd2d1f000..0xd2d1f970]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2f5518> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10410" daemon prio=10 tid=0x02045fd0 nid=0x3855 in Object.wait() [0xd400f000..0xd400fbf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd329a88> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10408" daemon prio=10 tid=0x01169528 nid=0x3853 in Object.wait() [0xd2fef000..0xd2fefaf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2f2b78> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10405" daemon prio=10 tid=0x0323d730 nid=0x3850 in Object.wait() [0xd3a8f000..0xd3a8fa70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2de280> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10404" daemon prio=10 tid=0x02fc2368 nid=0x384f in Object.wait() [0xd71df000..0xd71df8f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd311618> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10400" daemon prio=10 tid=0x029fc3e0 nid=0x384b in Object.wait() [0xd47af000..0xd47afaf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2b6d38> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10399" daemon prio=10 tid=0x03773890 nid=0x384a in Object.wait() [0xd34ef000..0xd34efb70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd3338f0> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10398" daemon prio=10 tid=0x0083ae50 nid=0x3849 in Object.wait() [0xd344f000..0xd344f9f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2ab378> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10396" daemon prio=10 tid=0x0099cf58 nid=0x3847 in Object.wait() [0xd277f000..0xd277f8f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2dbb38> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10395" daemon prio=10 tid=0x0099cd90 nid=0x3846 in Object.wait() [0xd506f000..0xd506f970]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2e9890> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10393" daemon prio=10 tid=0x0083ac88 nid=0x3844 in Object.wait() [0xd457f000..0xd457fc70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2f98a0> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10392" daemon prio=10 tid=0x00cf38d0 nid=0x3843 in Object.wait() [0xd461f000..0xd461faf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd2f9910> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10391" daemon prio=10 tid=0x01843dd0 nid=0x3842 in Object.wait() [0xd218f000..0xd218fb70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd24a230> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10390" daemon prio=10 tid=0x0226e270 nid=0x3841 in Object.wait() [0xd470f000..0xd470f9f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd319920> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10388" daemon prio=10 tid=0x0226e438 nid=0x383f in Object.wait() [0xd376f000..0xd376f8f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd3013f8> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10386" daemon prio=10 tid=0x00f57c38 nid=0x383d in Object.wait() [0xd3a3f000..0xd3a3fbf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0xcd32d908> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10382" daemon prio=10 tid=0x022fb2c8 nid=0x3839 in Object.wait() [0xd7bef000..0xd7bef9f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74003bb8> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3516" daemon prio=10 tid=0x003f3480 nid=0x3834 waiting on condition [0xd1d7d000..0xd1d7fc70]
                         at java.lang.Thread.sleep(Native Method)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.executeConfigFetchCommand(ConfigCollector.java:240)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.getConfiguration(ConfigCollector.java:150)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigBackupService.onConfigChangeEvent(ConfigBackupService.java:410)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ArchiveConfigBean.onConfigChangeEvent(ArchiveConfigBean.java:53)
                         at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                         at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                         at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
                         at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
                         at $Proxy301.onConfigChangeEvent(Unknown Source)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.sendEvent(ConfigChangeListenerMDB.java:160)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.onMessage(ConfigChangeListenerMDB.java:84)
                         at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPolicy.invokeInNoTx(TxPolicy.java:66)
                         at org.jboss.aspects.tx.TxInterceptor$NotSupported.invoke(TxInterceptor.java:112)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy294.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                         at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10376" daemon prio=10 tid=0x03384c68 nid=0x3832 in Object.wait() [0xd4acf000..0xd4acfb70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74055bf8> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10374" daemon prio=10 tid=0x02fcadb0 nid=0x3830 in Object.wait() [0xd3d5f000..0xd3d5fa70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74040278> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10371" daemon prio=10 tid=0x003f32b8 nid=0x382d in Object.wait() [0xd39ef000..0xd39efbf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74003928> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10370" daemon prio=10 tid=0x02c1cd58 nid=0x382c in Object.wait() [0xd75ef000..0xd75efc70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x740461a0> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10369" daemon prio=10 tid=0x02ba4600 nid=0x382b in Object.wait() [0xd1f5f000..0xd1f5faf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74026340> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10368" daemon prio=10 tid=0x01bb6020 nid=0x382a in Object.wait() [0xd209f000..0xd209fb70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73fb9c50> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10366" daemon prio=10 tid=0x00ebf988 nid=0x3828 in Object.wait() [0xd524f000..0xd524fa70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73fdf7d8> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10365" daemon prio=10 tid=0x003f1d68 nid=0x3827 in Object.wait() [0xd4fcf000..0xd4fcf8f0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x74033250> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10364" daemon prio=10 tid=0x03e6cd68 nid=0x3826 in Object.wait() [0xd1d2f000..0xd1d2f970]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73ff0678> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10361" daemon prio=10 tid=0x03631480 nid=0x3823 in Object.wait() [0xd43ef000..0xd43efaf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73fed738> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10355" daemon prio=10 tid=0x00837e08 nid=0x381c in Object.wait() [0xd42ff000..0xd42ffc70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73fb3dd0> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10354" daemon prio=10 tid=0x03e6cba0 nid=0x381b in Object.wait() [0xd3e9f000..0xd3e9faf0]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73f6f900> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "Thread-10353" daemon prio=10 tid=0x00513f00 nid=0x381a in Object.wait() [0xd27cf000..0xd27cfb70]
                         at java.lang.Object.wait(Native Method)
                         at java.lang.Object.wait(Object.java:474)
                         at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                         - locked <0x73f7a920> (a java.lang.Object)
                         at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                         at java.lang.Thread.run(Thread.java:595)
                        
                        "WorkManager(2)-3515" daemon prio=10 tid=0x021bf5d0 nid=0x3817 waiting on condition [0xd1ebd000..0xd1ebf8f0]
                         at java.lang.Thread.sleep(Native Method)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.executeConfigFetchCommand(ConfigCollector.java:240)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigCollector.getConfiguration(ConfigCollector.java:150)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigBackupService.onConfigChangeEvent(ConfigBackupService.java:410)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ArchiveConfigBean.onConfigChangeEvent(ArchiveConfigBean.java:53)
                         at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                         at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                         at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                         at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
                         at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
                         at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
                         at $Proxy301.onConfigChangeEvent(Unknown Source)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.sendEvent(ConfigChangeListenerMDB.java:160)
                         at com.cisco.nm.cmp.ems.configarchive.ejb.backupandrestore.ConfigChangeListenerMDB.onMessage(ConfigChangeListenerMDB.java:84)
                         at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
                         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                         at java.lang.reflect.Method.invoke(Method.java:585)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                         at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                         at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.aspects.tx.TxPolicy.invokeInNoTx(TxPolicy.java:66)
                         at org.jboss.aspects.tx.TxInterceptor$NotSupported.invoke(TxInterceptor.java:112)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                         at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                         at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                         at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                         at $Proxy294.onMessage(Unknown Source)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                         at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                         at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                         at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                         at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                         at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                         at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                         at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                         at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                         at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrap


                        • 9. Re: BisocketClientInvoker deadlock
                          ykonrad



                          I removed the irrelevant deployment from jboss deploy folder in order to minimize this stack trace. Here is the new minimized Tread dump

                          "RMI TCP Connection(40)-64.103.124.101" daemon prio=10 tid=0x0270d9f0 nid=0xef runnable [0xd981f000..0xd981f8f0]
                          at java.net.SocketInputStream.socketRead0(Native Method)
                          at java.net.SocketInputStream.read(SocketInputStream.java:129)
                          at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                          at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                          - locked <0xaa5d74d0> (a java.io.BufferedInputStream)
                          at java.io.FilterInputStream.read(FilterInputStream.java:66)
                          at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
                          at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
                          at java.lang.Thread.run(Thread.java:595)

                          "WorkerThread#25[64.103.124.218:2529]" prio=10 tid=0x0256ab40 nid=0xed waiting for monitor entry [0xd6e3d000..0xd6e3fbf0]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.deregister(IMOManagerBean.java:590)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#24[64.103.124.247:20610]" prio=10 tid=0x01c175d8 nid=0xeb waiting for monitor entry [0xd96dd000..0xd96dfaf0]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                          at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#23[64.103.124.247:20615]" prio=10 tid=0x01c17390 nid=0xea waiting for monitor entry [0xd97cd000..0xd97cfb70]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                          at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#22[64.103.124.247:20607]" prio=10 tid=0x022ae9c0 nid=0xe8 waiting for monitor entry [0xd972d000..0xd972fa70]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                          at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#21[64.103.124.247:20606]" prio=10 tid=0x02725528 nid=0xe7 waiting for monitor entry [0xd68eb000..0xd68ef8f0]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:1050)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.get(IMOManagerBean.java:336)
                          at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:79)
                          at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:70)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:103)
                          at $Proxy133.get(Unknown Source)
                          at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKeyIMO(RegistryCommandBean.java:934)
                          at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKey(RegistryCommandBean.java:64)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at com.cisco.nm.cmp.interceptors.AuthorizationInterceptor.authorize(AuthorizationInterceptor.java:68)
                          at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                          at com.cisco.nm.cmp.interceptors.AuditInterceptor.intercept(AuditInterceptor.java:50)
                          at com.cisco.nm.cmp.interceptors.BaseInterceptor.interceptWrapper(BaseInterceptor.java:17)
                          at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#20[64.103.124.247:20603]" prio=10 tid=0x027257c0 nid=0xe6 waiting for monitor entry [0xd98bb000..0xd98bf970]
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                          at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.(GWConnectionImpl.java:69)
                          at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                          at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:1050)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.get(IMOManagerBean.java:336)
                          at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:79)
                          at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:70)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:103)
                          at $Proxy133.get(Unknown Source)
                          at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKeyIMO(RegistryCommandBean.java:934)
                          at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKey(RegistryCommandBean.java:64)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at com.cisco.nm.cmp.interceptors.AuthorizationInterceptor.authorize(AuthorizationInterceptor.java:68)
                          at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                          at com.cisco.nm.cmp.interceptors.AuditInterceptor.intercept(AuditInterceptor.java:50)
                          at com.cisco.nm.cmp.interceptors.BaseInterceptor.interceptWrapper(BaseInterceptor.java:17)
                          at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "Thread-88" prio=10 tid=0x01c18698 nid=0xd4 in Object.wait() [0xd990f000..0xd990fc70]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x47425af0> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0x47425af0> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "WorkManager(2)-10" daemon prio=10 tid=0x02d1e5a8 nid=0xc7 in Object.wait() [0xd6dea000..0xd6def8f0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb94ac400> (a java.util.HashSet)
                          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
                          - locked <0xb94ac400> (a java.util.HashSet)
                          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
                          at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
                          at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
                          at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                          at org.jboss.remoting.Client.invoke(Client.java:1634)
                          at org.jboss.remoting.Client.invoke(Client.java:548)
                          at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                          at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                          at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                          at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
                          at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
                          - locked <0x4ba02568> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
                          at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                          - locked <0x4ba347f0> (a java.lang.Object)
                          at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                          at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                          at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                          at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                          at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                          at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                          - locked <0x4ba340f8> (a java.lang.Object)
                          at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                          at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                          at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                          at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                          at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                          at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                          at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                          at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                          at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                          at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                          at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                          at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                          at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                          at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                          at org.jboss.remoting.Client.invoke(Client.java:1634)
                          at org.jboss.remoting.Client.invoke(Client.java:548)
                          at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                          at org.jboss.remoting.Client.invokeOneway(Client.java:786)
                          at org.jboss.remoting.Client.invokeOneway(Client.java:776)
                          at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
                          at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                          at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                          at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                          at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                          at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                          at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
                          at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                          at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                          at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                          at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                          at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                          at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                          at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                          at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134)
                          at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141)
                          at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                          at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                          at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                          at $Proxy171.onMessage(Unknown Source)
                          at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                          at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                          at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                          at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                          at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                          at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                          at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                          at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                          at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                          at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-87" prio=10 tid=0x023abb10 nid=0xc2 in Object.wait() [0xfde8f000..0xfde8fb70]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x472adf38> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0x472adf38> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "WorkerThread#19[64.103.124.247:20550]" prio=10 tid=0x0314d308 nid=0xc0 waiting for monitor entry [0xd99fd000..0xd99ffa70]
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.asyncExecuteAndRegister(GWConnectionImpl.java:162)
                          at com.cisco.nm.cmp.imomanager.IMOManagerBean.asyncExecuteAndRegister(IMOManagerBean.java:790)
                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                          at java.lang.reflect.Method.invoke(Method.java:585)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                          at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                          at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                          at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                          at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                          at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                          at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                          at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                          at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "Thread-79" prio=10 tid=0x018eef28 nid=0xbf in Object.wait() [0xd9a4f000..0xd9a4f8f0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb9787390> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb9787390> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-80" prio=10 tid=0x018089f8 nid=0xbe in Object.wait() [0xd9a9f000..0xd9a9f970]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb95ee408> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb95ee408> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-85" prio=10 tid=0x02102ce8 nid=0xbd in Object.wait() [0xd9aef000..0xd9aefbf0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb977f410> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb977f410> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-81" prio=10 tid=0x01628238 nid=0xbc in Object.wait() [0xd9b3f000..0xd9b3fc70]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb966d268> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb966d268> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-82" prio=10 tid=0x02d59588 nid=0xbb in Object.wait() [0xd9b8f000..0xd9b8faf0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb969d308> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb969d308> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-83" prio=10 tid=0x01d62ca0 nid=0xba in Object.wait() [0xd9bdf000..0xd9bdfb70]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb977fce0> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb977fce0> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-84" prio=10 tid=0x00839e98 nid=0xb9 in Object.wait() [0xd9c2f000..0xd9c2f9f0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0xb9621520> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0xb9621520> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "WorkerThread#18[64.103.124.247:20601]" prio=10 tid=0x01e83e08 nid=0xb8 waiting for monitor entry [0xd737f000..0xd737fa70]
                          at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                          - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                          at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                          at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                          at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "WorkerThread#17[64.103.124.247:20604]" prio=10 tid=0x01649c70 nid=0xb7 waiting for monitor entry [0xd6d4f000..0xd6d4f8f0]
                          at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                          - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                          at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                          at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                          at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

                          "Thread-76" daemon prio=10 tid=0x02170400 nid=0xac in Object.wait() [0xda21f000..0xda21fc70]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x4e20c1f8> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0x4e20c1f8> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "Thread-75" daemon prio=10 tid=0x028261e8 nid=0xab in Object.wait() [0xd6e8f000..0xd6e8faf0]
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x4e262388> (a java.lang.Object)
                          at java.lang.Object.wait(Object.java:474)
                          at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                          - locked <0x4e262388> (a java.lang.Object)
                          at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                          at java.lang.Thread.run(Thread.java:595)

                          "WorkerThread#16[64.103.124.101:4330]" prio=10 tid=0x02eb8fa0 nid=0xa9 waiting for monitor entry [0xd65cd000..0xd65cf9f0]
                          at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                          - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                          at co

                          • 10. Re: BisocketClientInvoker deadlock
                            ykonrad


                            Thanks Tim,

                            For some reason I can't post the entire stack to the page (maximum message length)

                            Can I send the text file to you?

                            • 11. Re: BisocketClientInvoker deadlock
                              azafran

                              Full stack, part 1

                              Full thread dump Java HotSpot(TM) Server VM (1.5.0_08-b03 mixed mode):
                              
                              "RMI TCP Connection(40)-64.103.124.101" daemon prio=10 tid=0x0270d9f0 nid=0xef runnable [0xd981f000..0xd981f8f0]
                               at java.net.SocketInputStream.socketRead0(Native Method)
                               at java.net.SocketInputStream.read(SocketInputStream.java:129)
                               at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                               at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                               - locked <0xaa5d74d0> (a java.io.BufferedInputStream)
                               at java.io.FilterInputStream.read(FilterInputStream.java:66)
                               at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
                               at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
                               at java.lang.Thread.run(Thread.java:595)
                              
                              "WorkerThread#25[64.103.124.218:2529]" prio=10 tid=0x0256ab40 nid=0xed waiting for monitor entry [0xd6e3d000..0xd6e3fbf0]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.deregister(IMOManagerBean.java:590)
                               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "WorkerThread#24[64.103.124.247:20610]" prio=10 tid=0x01c175d8 nid=0xeb waiting for monitor entry [0xd96dd000..0xd96dfaf0]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                               at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "WorkerThread#23[64.103.124.247:20615]" prio=10 tid=0x01c17390 nid=0xea waiting for monitor entry [0xd97cd000..0xd97cfb70]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                               at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "WorkerThread#22[64.103.124.247:20607]" prio=10 tid=0x022ae9c0 nid=0xe8 waiting for monitor entry [0xd972d000..0xd972fa70]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                               at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "WorkerThread#21[64.103.124.247:20606]" prio=10 tid=0x02725528 nid=0xe7 waiting for monitor entry [0xd68eb000..0xd68ef8f0]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.removeListenerFromMap(SimpleUserConnectionImpl.java:158)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:139)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:1050)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.get(IMOManagerBean.java:336)
                               at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:79)
                               at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:70)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:103)
                               at $Proxy133.get(Unknown Source)
                               at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKeyIMO(RegistryCommandBean.java:934)
                               at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKey(RegistryCommandBean.java:64)
                               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at com.cisco.nm.cmp.interceptors.AuthorizationInterceptor.authorize(AuthorizationInterceptor.java:68)
                               at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                               at com.cisco.nm.cmp.interceptors.AuditInterceptor.intercept(AuditInterceptor.java:50)
                               at com.cisco.nm.cmp.interceptors.BaseInterceptor.interceptWrapper(BaseInterceptor.java:17)
                               at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "WorkerThread#20[64.103.124.247:20603]" prio=10 tid=0x027257c0 nid=0xe6 waiting for monitor entry [0xd98bb000..0xd98bf970]
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                               - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                               at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                               at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                               at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                               at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                               at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:819)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:1050)
                               at com.cisco.nm.cmp.imomanager.IMOManagerBean.get(IMOManagerBean.java:336)
                               at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.ejb3.remoting.IsLocalInterceptor.invokeLocal(IsLocalInterceptor.java:79)
                               at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:70)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:103)
                               at $Proxy133.get(Unknown Source)
                               at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKeyIMO(RegistryCommandBean.java:934)
                               at com.cisco.nm.cmp.gw.services.registrycommand.RegistryCommandBean.getKey(RegistryCommandBean.java:64)
                               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at com.cisco.nm.cmp.interceptors.AuthorizationInterceptor.authorize(AuthorizationInterceptor.java:68)
                               at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                               at com.cisco.nm.cmp.interceptors.AuditInterceptor.intercept(AuditInterceptor.java:50)
                               at com.cisco.nm.cmp.interceptors.BaseInterceptor.interceptWrapper(BaseInterceptor.java:17)
                               at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                               at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                               at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                               at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                               at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                               at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                               at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                               at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                              
                              "Thread-88" prio=10 tid=0x01c18698 nid=0xd4 in Object.wait() [0xd990f000..0xd990fc70]
                               at java.lang.Object.wait(Native Method)
                               - waiting on <0x47425af0> (a java.lang.Object)
                               at java.lang.Object.wait(Object.java:474)
                               at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                               - locked <0x47425af0> (a java.lang.Object)
                               at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                               at java.lang.Thread.run(Thread.java:595)
                              
                              "WorkManager(2)-10" daemon prio=10 tid=0x02d1e5a8 nid=0xc7 in Object.wait() [0xd6dea000..0xd6def8f0]
                               at java.lang.Object.wait(Native Method)
                               - waiting on <0xb94ac400> (a java.util.HashSet)
                               at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
                               - locked <0xb94ac400> (a java.util.HashSet)
                               at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
                               at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
                               at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
                               at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                               at org.jboss.remoting.Client.invoke(Client.java:1634)
                               at org.jboss.remoting.Client.invoke(Client.java:548)
                               at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                               at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
                               at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
                               at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
                               at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
                               - locked <0x4ba02568> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
                               at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
                               - locked <0x4ba347f0> (a java.lang.Object)
                               at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
                               at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
                               at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
                               at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
                               at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
                               at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
                               - locked <0x4ba340f8> (a java.lang.Object)
                               at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
                               at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                               at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                               at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                               at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                               at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                               at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                               at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                               at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                               at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                               at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                               at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
                               at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
                               at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
                               at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
                               at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                               at org.jboss.remoting.Client.invoke(Client.java:1634)
                               at org.jboss.remoting.Client.invoke(Client.java:548)
                               at org.jboss.remoting.Client.invokeOneway(Client.java:598)
                               at org.jboss.remoting.Client.invokeOneway(Client.java:786)
                               at org.jboss.remoting.Client.invokeOneway(Client.java:776)
                               at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
                               at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:473)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                               at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                               at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                               at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                               at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                               at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                               at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
                               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                               at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                               at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                               at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                               at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                               at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134)
                               at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141)
                               at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117)
                               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                               at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                               at java.lang.reflect.Method.invoke(Method.java:585)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                               at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                               at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                               at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                               at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                               at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
                               at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
                               at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
                               at $Proxy171.onMessage(Unknown Source)
                               at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
                               at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:159)
                               at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:802)
                               at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                               at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                               at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
                               at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
                               at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
                               at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
                               at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
                               at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
                               at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                               at java.lang.Thread.run(Thread.java:595)
                              


                              • 12. Re: BisocketClientInvoker deadlock
                                azafran

                                Full stack, part 2

                                "Thread-87" prio=10 tid=0x023abb10 nid=0xc2 in Object.wait() [0xfde8f000..0xfde8fb70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x472adf38> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x472adf38> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "WorkerThread#19[64.103.124.247:20550]" prio=10 tid=0x0314d308 nid=0xc0 waiting for monitor entry [0xd99fd000..0xd99ffa70]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.asyncExecuteAndRegister(GWConnectionImpl.java:162)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.asyncExecuteAndRegister(IMOManagerBean.java:790)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "Thread-79" prio=10 tid=0x018eef28 nid=0xbf in Object.wait() [0xd9a4f000..0xd9a4f8f0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb9787390> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb9787390> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-80" prio=10 tid=0x018089f8 nid=0xbe in Object.wait() [0xd9a9f000..0xd9a9f970]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb95ee408> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb95ee408> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-85" prio=10 tid=0x02102ce8 nid=0xbd in Object.wait() [0xd9aef000..0xd9aefbf0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb977f410> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb977f410> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-81" prio=10 tid=0x01628238 nid=0xbc in Object.wait() [0xd9b3f000..0xd9b3fc70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb966d268> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb966d268> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-82" prio=10 tid=0x02d59588 nid=0xbb in Object.wait() [0xd9b8f000..0xd9b8faf0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb969d308> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb969d308> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-83" prio=10 tid=0x01d62ca0 nid=0xba in Object.wait() [0xd9bdf000..0xd9bdfb70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb977fce0> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb977fce0> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-84" prio=10 tid=0x00839e98 nid=0xb9 in Object.wait() [0xd9c2f000..0xd9c2f9f0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0xb9621520> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0xb9621520> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "WorkerThread#18[64.103.124.247:20601]" prio=10 tid=0x01e83e08 nid=0xb8 waiting for monitor entry [0xd737f000..0xd737fa70]
                                 at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                                 - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                                 at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                                 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                                 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#17[64.103.124.247:20604]" prio=10 tid=0x01649c70 nid=0xb7 waiting for monitor entry [0xd6d4f000..0xd6d4f8f0]
                                 at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                                 - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                                 at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                                 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                                 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "Thread-76" daemon prio=10 tid=0x02170400 nid=0xac in Object.wait() [0xda21f000..0xda21fc70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4e20c1f8> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4e20c1f8> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-75" daemon prio=10 tid=0x028261e8 nid=0xab in Object.wait() [0xd6e8f000..0xd6e8faf0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4e262388> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4e262388> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "WorkerThread#16[64.103.124.101:4330]" prio=10 tid=0x02eb8fa0 nid=0xa9 waiting for monitor entry [0xd65cd000..0xd65cf9f0]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeAndRegister(GWConnectionImpl.java:252)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:669)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:699)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#15[64.103.124.101:4386]" prio=10 tid=0x01a31da0 nid=0xa8 waiting for monitor entry [0xd661d000..0xd661fa70]
                                 at com.cisco.nm.cmp.imobridge.resources.SimpleGWCommandListenerImpl.stop(SimpleGWCommandListenerImpl.java:323)
                                 - waiting to lock <0xb9dfa8b0> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
                                 at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.deregister(SimpleUserConnectionImpl.java:273)
                                 - locked <0xaa2f9ab8> (a com.cisco.nm.cmp.gw.commandmanager.commands.RegistrationIdImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.deregister(GWConnectionImpl.java:213)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.deregister(IMOManagerBean.java:595)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#14[64.103.124.247:20551]" prio=10 tid=0x01a326e0 nid=0xa7 waiting for monitor entry [0xd666e000..0xd666f8f0]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.execute(GWConnectionImpl.java:225)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:820)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                                 at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#13[64.103.124.101:4356]" prio=10 tid=0x01affe98 nid=0xa6 waiting for monitor entry [0xd66bd000..0xd66bf970]
                                 at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:109)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.execute(SimpleUserConnectionImpl.java:176)
                                 at com.cisco.nm.cmp.imobridge.resources.SimpleUserConnectionImpl.getKey(SimpleUserConnectionImpl.java:356)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.<init>(GWConnectionImpl.java:69)
                                 at com.cisco.nm.cmp.gjc.GWManagedConnection.getConnection(GWManagedConnection.java:103)
                                 at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
                                 at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                                 at com.cisco.nm.cmp.gjc.GWConnectionFactoryImpl.getConnection(GWConnectionFactoryImpl.java:126)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.getGWConnection(IMOManagerBean.java:859)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:668)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:699)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#12[64.103.124.247:20549]" prio=10 tid=0x02ebc3b8 nid=0xa5 waiting for monitor entry [0xd670d000..0xd670fbf0]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.asyncExecuteAndRegister(GWConnectionImpl.java:162)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.asyncExecuteAndRegister(IMOManagerBean.java:790)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "Thread-67" prio=10 tid=0x02f37a30 nid=0xa4 in Object.wait() [0xd675f000..0xd675fc70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4ba33dc8> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4ba33dc8> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-63" prio=10 tid=0x02798988 nid=0xa3 in Object.wait() [0xd67af000..0xd67afaf0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4ba0ab88> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4ba0ab88> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-69" prio=10 tid=0x02076b20 nid=0xa2 in Object.wait() [0xd67ff000..0xd67ffb70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b9efd38> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4b9efd38> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-66" prio=10 tid=0x0329d9c0 nid=0xa1 in Object.wait() [0xd684f000..0xd684f9f0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b9f0830> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4b9f0830> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-65" prio=10 tid=0x031b3de0 nid=0xa0 in Object.wait() [0xd689f000..0xd689fa70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b9d4ba8> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4b9d4ba8> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "Thread-68" prio=10 tid=0x02ee8020 nid=0x9e in Object.wait() [0xd693f000..0xd693f970]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b9b41f8> (a java.lang.Object)
                                 at java.lang.Object.wait(Object.java:474)
                                 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                 - locked <0x4b9b41f8> (a java.lang.Object)
                                 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                 at java.lang.Thread.run(Thread.java:595)
                                
                                "WorkerThread#11[64.103.124.101:4347]" prio=10 tid=0x02ee0a30 nid=0x9d waiting for monitor entry [0xd698d000..0xd698fbf0]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeAndRegister(GWConnectionImpl.java:252)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:669)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.executeAndRegister(IMOManagerBean.java:699)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#10[64.103.124.247:20552]" prio=10 tid=0x01478828 nid=0x9c waiting for monitor entry [0xd69dd000..0xd69dfc70]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.asyncExecuteAndRegister(GWConnectionImpl.java:162)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.asyncExecuteAndRegister(IMOManagerBean.java:790)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#9[64.103.124.101:4342]" prio=10 tid=0x02306710 nid=0x9b waiting for monitor entry [0xd6a2e000..0xd6a2faf0]
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                 at com.cisco.nm.cmp.gjc.GWConnectionImpl.execute(GWConnectionImpl.java:225)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:820)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                                 at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                                 at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                 at java.lang.reflect.Method.invoke(Method.java:585)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                 at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#8[64.103.124.101:4216]" prio=10 tid=0x0203bff0 nid=0x9a in Object.wait() [0xd6a7f000..0xd6a7fb70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b533388> (a org.jboss.remoting.transport.socket.ServerThread)
                                 at java.lang.Object.wait(Object.java:474)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:212)
                                 - locked <0x4b533388> (a org.jboss.remoting.transport.socket.ServerThread)
                                
                                "WorkerThread#7[64.103.124.247:20461]" prio=10 tid=0x03220418 nid=0x99 in Object.wait() [0xd6acf000..0xd6acf9f0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b532fa8> (a org.jboss.remoting.transport.socket.ServerThread)
                                 at java.lang.Object.wait(Object.java:474)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:212)
                                 - locked <0x4b532fa8> (a org.jboss.remoting.transport.socket.ServerThread)
                                
                                "Timer-9" daemon prio=10 tid=0x02aa0e00 nid=0x98 in Object.wait() [0xd6b1f000..0xd6b1fa70]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4c245be8> (a java.util.TaskQueue)
                                 at java.util.TimerThread.mainLoop(Timer.java:509)
                                 - locked <0x4c245be8> (a java.util.TaskQueue)
                                 at java.util.TimerThread.run(Timer.java:462)
                                
                                "WorkerThread#6[64.103.124.101:4242]" prio=10 tid=0x027e43c0 nid=0x97 runnable [0xd6b6f000..0xd6b6f8f0]
                                 at java.net.SocketInputStream.socketRead0(Native Method)
                                 at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                                 - locked <0xb992aad8> (a java.io.BufferedInputStream)
                                 at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:823)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:509)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#5[64.103.124.101:4240]" prio=10 tid=0x0329fbe8 nid=0x96 runnable [0xd6bbf000..0xd6bbf970]
                                 at java.net.SocketInputStream.socketRead0(Native Method)
                                 at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                                 - locked <0xb98c96f8> (a java.io.BufferedInputStream)
                                 at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:823)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:509)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                
                                "WorkerThread#4[64.103.124.247:20459]" prio=10 tid=0x01a3c6b8 nid=0x95 in Object.wait() [0xd6c0f000..0xd6c0fbf0]
                                 at java.lang.Object.wait(Native Method)
                                 - waiting on <0x4b532bd0> (a org.jboss.remoting.transport.socket.ServerThread)
                                 at java.lang.Object.wait(Object.java:474)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:212)
                                 - locked <0x4b532bd0> (a org.jboss.remoting.transport.socket.ServerThread)
                                
                                "WorkerThread#3[64.103.124.247:20599]" prio=10 tid=0x02aa9f50 nid=0x94 waiting for monitor entry [0xd6c5f000..0xd6c5fc70]
                                 at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                                 - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                                 at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                                 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                                 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                


                                • 13. Re: BisocketClientInvoker deadlock
                                  azafran

                                  Full stack, part 3

                                  "Timer-8" daemon prio=10 tid=0x0238ff18 nid=0x93 in Object.wait() [0xd6caf000..0xd6cafaf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47080570> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
                                   - locked <0x47080570> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
                                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingInMemory(MessagingPostOffice.java:2280)
                                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalRemoveBinding(MessagingPostOffice.java:1902)
                                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBinding(MessagingPostOffice.java:464)
                                   at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:400)
                                   - locked <0x4b9a2598> (a java.util.HashSet)
                                   at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.closeConsumersForClientVMID(SimpleConnectionManager.java:377)
                                   - locked <0x47196dc8> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)
                                   at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(SimpleConnectionManager.java:173)
                                   - locked <0x47196dc8> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)
                                   at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(SimpleConnectionManager.java:200)
                                   at org.jboss.remoting.ConnectionNotifier.connectionLost(ConnectionNotifier.java:55)
                                   - locked <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                                   at org.jboss.remoting.Lease.notifyClientLost(Lease.java:211)
                                   at org.jboss.remoting.Lease.access$500(Lease.java:39)
                                   at org.jboss.remoting.Lease$LeaseTimerTask.run(Lease.java:249)
                                   at java.util.TimerThread.mainLoop(Timer.java:512)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "WorkerThread#2[64.103.124.247:20598]" prio=10 tid=0x01a70b18 nid=0x92 runnable [0xd6cff000..0xd6cffb70]
                                   at java.net.SocketInputStream.socketRead0(Native Method)
                                   at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                                   - locked <0xa3f47768> (a java.io.BufferedInputStream)
                                   at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                   at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:823)
                                   at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:509)
                                   at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                   at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                  
                                  "WorkerThread#1[64.103.124.247:20611]" prio=10 tid=0x0224b040 nid=0x90 runnable [0xd6d9f000..0xd6d9fa70]
                                   at java.net.SocketInputStream.socketRead0(Native Method)
                                   at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                                   at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
                                   - locked <0xa43d9ee0> (a java.io.BufferedInputStream)
                                   at java.io.FilterInputStream.read(FilterInputStream.java:66)
                                   at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:823)
                                   at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:509)
                                   at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
                                   at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                  
                                  "Thread-49" daemon prio=10 tid=0x0196d9b8 nid=0x8b waiting on condition [0xd6f2f000..0xd6f2faf0]
                                   at java.lang.Thread.sleep(Native Method)
                                   at com.cisco.nm.cmp.nbi.wsdm.platform.security.session.UserSessionMonitorThread.run(UserSessionMonitorThread.java:97)
                                  
                                  "http-0.0.0.0-8080-7" daemon prio=10 tid=0x01da1648 nid=0x8a in Object.wait() [0xd6f7f000..0xd6f7fb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0b88> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0b88> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-6" daemon prio=10 tid=0x00b97058 nid=0x89 in Object.wait() [0xd6fcf000..0xd6fcf9f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0bd0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0bd0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-5" daemon prio=10 tid=0x031620d0 nid=0x88 in Object.wait() [0xd701f000..0xd701fa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0b40> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0b40> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-4" daemon prio=10 tid=0x014cfec0 nid=0x87 in Object.wait() [0xd706f000..0xd706f8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0b58> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0b58> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-3" daemon prio=10 tid=0x02033e20 nid=0x86 in Object.wait() [0xd70bf000..0xd70bf970]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0ba0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0ba0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-2" daemon prio=10 tid=0x03048418 nid=0x85 in Object.wait() [0xd710f000..0xd710fbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0b70> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0b70> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-1" daemon prio=10 tid=0x00b965f0 nid=0x84 in Object.wait() [0xd715f000..0xd715fc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d3a0bb8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
                                   - locked <0x4d3a0bb8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-5" prio=10 tid=0x019db8f8 nid=0x81 waiting on condition [0xd71ff000..0xd71ff9f0]
                                   at java.lang.Thread.sleep(Native Method)
                                   at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:248)
                                   at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)
                                  
                                  "ajp-0.0.0.0-8009-Acceptor-0" daemon prio=10 tid=0x0256a7f8 nid=0x80 runnable [0xd724f000..0xd724fa70]
                                   at java.net.PlainSocketImpl.socketAccept(Native Method)
                                   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                   - locked <0x47b630b0> (a java.net.SocksSocketImpl)
                                   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                   at java.net.ServerSocket.accept(ServerSocket.java:421)
                                   at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "http-0.0.0.0-8080-Acceptor-0" daemon prio=10 tid=0x01a92f10 nid=0x7f runnable [0xd729f000..0xd729f8f0]
                                   at java.net.PlainSocketImpl.socketAccept(Native Method)
                                   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                   - locked <0x483f5c20> (a java.net.SocksSocketImpl)
                                   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                   at java.net.ServerSocket.accept(ServerSocket.java:421)
                                   at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
                                   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "JBossLifeThread" prio=10 tid=0x01773e08 nid=0x7e in Object.wait() [0xd72ef000..0xd72ef970]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4d9052d0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)
                                   - locked <0x4d9052d0> (a java.lang.Object)
                                  
                                  "Timer-7" daemon prio=10 tid=0x02163860 nid=0x7c in Object.wait() [0xd73cf000..0xd73cfc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bf50328> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x4bf50328> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "Thread-38" prio=10 tid=0x028346c0 nid=0x7b waiting on condition [0xd741f000..0xd741faf0]
                                   at java.lang.Thread.sleep(Native Method)
                                   at com.cisco.nm.cmp.nbi.wsdm.platform.background.BackgroundFileMonitorProcess$FileMonitorThread.run(BackgroundFileMonitorProcess.java:92)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Timer-6" daemon prio=10 tid=0x025a22f0 nid=0x7a in Object.wait() [0xd746f000..0xd746fb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bf43798> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x4bf43798> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "Thread-35" prio=10 tid=0x02697ea0 nid=0x79 waiting on condition [0xd75ef000..0xd75ef9f0]
                                   at java.lang.Thread.sleep(Native Method)
                                   at com.cisco.nm.cmp.nbi.wsdm.platform.background.RuntimeSystemPropertiesWatcher$ExternalSystemParamtersMonitorThread.run(RuntimeSystemPropertiesWatcher.java:81)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Timer-5" daemon prio=10 tid=0x01bf6038 nid=0x78 in Object.wait() [0xd77ef000..0xd77efa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bf3bf40> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x4bf3bf40> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "Connection Consumer for dest JBossQueue[NBIJobQueue] id=7" daemon prio=10 tid=0x015bdf98 nid=0x77 in Object.wait() [0xda16f000..0xda16f8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4b666f50> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x4b666f50> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-32" daemon prio=10 tid=0x00af5bd0 nid=0x76 in Object.wait() [0xda1bf000..0xda1bf970]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4b9c37b0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x4b9c37b0> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "QuartzScheduler_Scheduler1-1_MisfireHandler" prio=10 tid=0x026ce928 nid=0x74 waiting on condition [0xda32f000..0xda32fc70]
                                   at java.lang.Thread.sleep(Native Method)
                                   at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:3829)
                                  
                                  "Scheduler1_QuartzSchedulerThread" prio=10 tid=0x00aa00d0 nid=0x73 waiting on condition [0xdabef000..0xdabefaf0]
                                   at java.lang.Thread.sleep(Native Method)
                                   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:420)
                                  
                                  "Scheduler1_Worker-6" prio=10 tid=0x01958a20 nid=0x72 in Object.wait() [0xdaabf000..0xdaabfb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc7538> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc7538> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Scheduler1_Worker-5" prio=10 tid=0x014791d8 nid=0x71 in Object.wait() [0xdacbf000..0xdacbf9f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc7e28> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc7e28> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Scheduler1_Worker-4" prio=10 tid=0x01d09df8 nid=0x70 in Object.wait() [0xdab9f000..0xdab9fa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc7bd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc7bd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Scheduler1_Worker-3" prio=10 tid=0x0254ed50 nid=0x6f in Object.wait() [0xda9bf000..0xda9bf8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc76d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc76d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Scheduler1_Worker-2" prio=10 tid=0x01efdca8 nid=0x6e in Object.wait() [0xda96f000..0xda96f970]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc75d8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc75d8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Scheduler1_Worker-1" prio=10 tid=0x027a7218 nid=0x6d in Object.wait() [0xdc93f000..0xdc93fbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4bcc74b0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                   - locked <0x4bcc74b0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                  
                                  "Logger notifications thread" daemon prio=10 tid=0x02108c10 nid=0x6b in Object.wait() [0xda37f000..0xda37faf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4703af00> (a java.lang.Object)
                                   at com.cisco.nm.cmp.system.os.logger.AbstractLoggerImpl$1.run(AbstractLoggerImpl.java:333)
                                   - locked <0x4703af00> (a java.lang.Object)
                                  
                                  "Connection Consumer for dest JBossQueue[dmmNotifyRetrievedPageGroupsQueue] id=6" daemon prio=10 tid=0x01c9d6b0 nid=0x6a in Object.wait() [0xda3cf000..0xda3cfb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x473adfa0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x473adfa0> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Connection Consumer for dest JBossQueue[GWAuthorizationDataNotificationsQueue] id=5" daemon prio=10 tid=0x0060a8f0 nid=0x69 in Object.wait() [0xda41f000..0xda41f9f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47438108> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x47438108> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Connection Consumer for dest JBossQueue[dmmSynchronizedGroupsQueue] id=4" daemon prio=10 tid=0x022c38b8 nid=0x68 in Object.wait() [0xda46f000..0xda46fa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47425b98> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x47425b98> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Connection Consumer for dest JBossQueue[IMOMAsynchQueue] id=3" daemon prio=10 tid=0x01db34d0 nid=0x67 in Object.wait() [0xda4bf000..0xda4bf8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47391b70> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x47391b70> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-27" daemon prio=10 tid=0x01cd5750 nid=0x66 in Object.wait() [0xda50f000..0xda50f970]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x4761a708> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x4761a708> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Connection Consumer for dest JBossQueue[dmmSynchronizedPageGroupsQueue] id=1" daemon prio=10 tid=0x002aef98 nid=0x65 in Object.wait() [0xda55f000..0xda55fbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x472adba0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x472adba0> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Connection Consumer for dest JBossQueue[GatewayNotificationsPrivateQueue] id=2" daemon prio=10 tid=0x01c77cb8 nid=0x64 in Object.wait() [0xda5af000..0xda5afc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x473e4240> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:799)
                                   at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:427)
                                   - locked <0x473e4240> (a java.lang.Object)
                                   at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect52.invoke(ConsumerAspect52.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
                                   at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-29" daemon prio=10 tid=0x008cc200 nid=0x63 in Object.wait() [0xda5ff000..0xda5ffaf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x473706c0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x473706c0> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-24" daemon prio=10 tid=0x02c57b20 nid=0x62 in Object.wait() [0xda64f000..0xda64fb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x476343d8> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x476343d8> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-28" daemon prio=10 tid=0x0151f188 nid=0x61 in Object.wait() [0xda69f000..0xda69f9f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x471863a8> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x471863a8> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-26" daemon prio=10 tid=0x0292a868 nid=0x60 in Object.wait() [0xda6ef000..0xda6efa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47746698> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x47746698> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-25" daemon prio=10 tid=0x01db3cc8 nid=0x5f in Object.wait() [0xda73f000..0xda73f8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x476c3cc0> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x476c3cc0> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "UserConnectionPollerThread" prio=10 tid=0x0296e530 nid=0x5e waiting for monitor entry [0xda8cf000..0xda8cf970]
                                   at com.cisco.nm.cmp.gjc.cm.UserConnectionManager.expireConnections(UserConnectionManager.java:278)
                                   - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                   at com.cisco.nm.cmp.gjc.cm.UserConnectionPoller.run(UserConnectionPoller.java:62)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Notify Thread" prio=10 tid=0x01576108 nid=0x5d in Object.wait() [0xda78d000..0xda78fbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47080560> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
                                   - locked <0x47080560> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
                                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
                                   at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
                                   at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
                                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
                                   at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
                                   at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                                   at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
                                   at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
                                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                   at java.lang.reflect.Method.invoke(Method.java:585)
                                   at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                                   at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
                                   at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                                   at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
                                   at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                                   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                   at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                                   at org.jboss.remoting.Client.invoke(Client.java:1634)
                                   at org.jboss.remoting.Client.invoke(Client.java:548)
                                   at org.jboss.remoting.Client.invoke(Client.java:536)
                                   at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                                   at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                   at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                   at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
                                   at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
                                   at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
                                   at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                                   at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                                   at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
                                   at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
                                   at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
                                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
                                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
                                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
                                   at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
                                   at com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:324)
                                   - locked <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                   at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)
                                   - locked <0xb9dfa8b0> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
                                   at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)
                                   - locked <0xb9dfa8b0> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
                                   at com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)
                                   at com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)
                                   at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)
                                   at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)
                                  
                                  "Connection State" prio=10 tid=0x01bd09f8 nid=0x5c in Object.wait() [0xda7df000..0xda7dfc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x471a7d40> (a com.cisco.nm.cmp.gw.adapters.ptp.protocol.ConnectionState)
                                   at java.lang.Object.wait(Object.java:474)
                                   at com.cisco.nm.cmp.gw.ptpclient.protocol.ConnectionStateThread.run(ConnectionStateThread.java:40)
                                   - locked <0x471a7d40> (a com.cisco.nm.cmp.gw.adapters.ptp.protocol.ConnectionState)
                                  
                                  "Recieve Thread" prio=10 tid=0x022f10d8 nid=0x5b runnable [0xda82f000..0xda82faf0]
                                   at java.net.SocketInputStream.socketRead0(Native Method)
                                   at java.net.SocketInputStream.read(SocketInputStream.java:129)
                                   at java.net.SocketInputStream.read(SocketInputStream.java:182)
                                   at java.io.DataInputStream.readInt(DataInputStream.java:353)
                                   at com.cisco.nm.cmp.gw.adapters.ptp.protocol.SocketConnection.receive(SocketConnection.java:93)
                                   at com.cisco.nm.cmp.gw.adapters.ptp.protocol.SocketConnection.receive(SocketConnection.java:85)
                                   at com.cisco.nm.cmp.gw.ptpclient.protocol.ReceiveThread.run(ReceiveThread.java:71)
                                  
                                  "Send Thread" prio=10 tid=0x034e23c8 nid=0x5a in Object.wait() [0xda87f000..0xda87fb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x471a7d08> (a com.cisco.nm.cmp.gw.adapters.ptp.util.BlockingQueue)
                                   at com.cisco.nm.cmp.gw.adapters.ptp.util.BlockingQueue.dequeue(BlockingQueue.java:57)
                                   - locked <0x471a7d08> (a com.cisco.nm.cmp.gw.adapters.ptp.util.BlockingQueue)
                                   at com.cisco.nm.cmp.gw.ptpclient.protocol.SendThread.run(SendThread.java:54)
                                  
                                  "IdleRemover" daemon prio=10 tid=0x02af17e0 nid=0x57 in Object.wait() [0xda91f000..0xda91f8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47e906b0> (a java.util.ArrayList)
                                   at org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)
                                   - locked <0x47e906b0> (a java.util.ArrayList)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Timer-4" daemon prio=10 tid=0x00b6ebf0 nid=0x54 in Object.wait() [0xdaa0f000..0xdaa0fc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47838418> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x47838418> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "Dummy Log processing thread" daemon prio=10 tid=0x0213a028 nid=0x52 in Object.wait() [0xdab3f000..0xdab3fb70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x47174c68> (a java.lang.Object)
                                   at com.cisco.nm.cmp.system.os.logger.DummyLogger$1.run(DummyLogger.java:163)
                                   - locked <0x47174c68> (a java.lang.Object)
                                  
                                  "EJB-Timer-6[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmQueueNotificationEJB]" prio=10 tid=0x008713e0 nid=0x4e waiting for monitor entry [0xdad5d000..0xdad5f970]
                                   at java.util.Collections$SynchronizedList.add(Collections.java:1826)
                                   - waiting to lock <0x47148630> (a java.util.Collections$SynchronizedRandomAccessList)
                                   at org.jboss.remoting.ConnectionNotifier.addListenerFirst(ConnectionNotifier.java:29)
                                   at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1490)
                                   at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:641)
                                   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:796)
                                   at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
                                   at org.jboss.remoting.Client.invoke(Client.java:1634)
                                   at org.jboss.remoting.Client.addCallbackListener(Client.java:1703)
                                   at org.jboss.remoting.Client.addListener(Client.java:921)
                                   at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:230)
                                   at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:340)
                                   at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
                                   at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                                   at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
                                   at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
                                   at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
                                   at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
                                   at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
                                   at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
                                   at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:82)
                                   at com.cisco.nm.dmm.ejb.QueueNotificationTimerBean.timeoutHandler(QueueNotificationTimerBean.java:166)
                                   at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
                                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                   at java.lang.reflect.Method.invoke(Method.java:585)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                   at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                   at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
                                   at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                   at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                   at org.jboss.ejb3.stateless.StatelessContainer.callTimeout(StatelessContainer.java:151)
                                   at org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
                                   at java.util.TimerThread.mainLoop(Timer.java:512)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "EJB-Timer-5[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmNotificationEJB]" prio=10 tid=0x0171a098 nid=0x4d in Object.wait() [0xdac3f000..0xdac3fbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x479e19d8> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x479e19d8> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "EJB-Timer-4[target=jboss.j2ee:service=EJB3,ear=dmm.ear.ear,jar=dmm.ejb-1.0.jar,name=dmmSyncAuthEJB]" prio=10 tid=0x01c24d80 nid=0x4c in Object.wait() [0xdad0f000..0xdad0fc70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x479da8c8> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x479da8c8> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "Thread-14" prio=10 tid=0x00632990 nid=0x48 in Object.wait() [0xdadaf000..0xdadafa70]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x46e286a0> (a org.apache.xmlrpc.WebServer$Runner)
                                   at java.lang.Object.wait(Object.java:474)
                                   at org.apache.xmlrpc.WebServer$Runner.run(Unknown Source)
                                   - locked <0x46e286a0> (a org.apache.xmlrpc.WebServer$Runner)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Timer-3" prio=10 tid=0x001f7558 nid=0x47 in Object.wait() [0xdaf1f000..0xdaf1f8f0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x46e49340> (a java.util.TaskQueue)
                                   at java.util.TimerThread.mainLoop(Timer.java:509)
                                   - locked <0x46e49340> (a java.util.TaskQueue)
                                   at java.util.TimerThread.run(Timer.java:462)
                                  
                                  "XML-RPC Weblistener" prio=10 tid=0x01db1658 nid=0x46 runnable [0xdaf6f000..0xdaf6f970]
                                   at java.net.PlainSocketImpl.socketAccept(Native Method)
                                   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                   - locked <0x46e284f0> (a java.net.SocksSocketImpl)
                                   at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                   at java.net.ServerSocket.accept(ServerSocket.java:421)
                                   at org.apache.xmlrpc.WebServer.run(Unknown Source)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-12" prio=10 tid=0x01edaf20 nid=0x45 in Object.wait() [0xdafbf000..0xdafbfbf0]
                                   at java.lang.Object.wait(Native Method)
                                   - waiting on <0x474aba60> (a java.lang.Object)
                                   at java.lang.Object.wait(Object.java:474)
                                   at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
                                   - locked <0x474aba60> (a java.lang.Object)
                                   at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
                                   at java.lang.Thread.run(Thread.java:595)
                                  
                                  "Thread-11" daemon prio=10 tid=0x02c584e0 nid=0x44 waiting on condition [0xdb00f000..0xdb00fc70]
                                   at java.lang.Thread.sleep(Native Method)
                                   at org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)
                                   at java.lang.Thread.run(Thread.java:595)
                                  



                                  • 14. Re: BisocketClientInvoker deadlock
                                    azafran

                                    Full stack, part 4 and final

                                    "Thread-10" daemon prio=10 tid=0x02fb9a40 nid=0x43 in Object.wait() [0xdb05f000..0xdb05faf0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x47ea3eb8> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
                                     at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)
                                     - locked <0x47ea3eb8> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
                                    
                                    "JCA PoolFiller" prio=10 tid=0x00aa0428 nid=0x42 in Object.wait() [0xdb0af000..0xdb0afb70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x48565a78> (a java.util.LinkedList)
                                     at java.lang.Object.wait(Object.java:474)
                                     at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:87)
                                     - locked <0x48565a78> (a java.util.LinkedList)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "Timer-2" daemon prio=10 tid=0x023fb3c8 nid=0x41 in Object.wait() [0xdb0ff000..0xdb0ff9f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x47027a90> (a java.util.TaskQueue)
                                     at java.util.TimerThread.mainLoop(Timer.java:509)
                                     - locked <0x47027a90> (a java.util.TaskQueue)
                                     at java.util.TimerThread.run(Timer.java:462)
                                    
                                    "DefaultQuartzScheduler_QuartzSchedulerThread" prio=10 tid=0x01c41ff0 nid=0x40 waiting on condition [0xdb14f000..0xdb14fa70]
                                     at java.lang.Thread.sleep(Native Method)
                                     at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:420)
                                    
                                    "DefaultQuartzScheduler_Worker-10" prio=10 tid=0x019593d8 nid=0x3f in Object.wait() [0xdb19f000..0xdb19f8f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f3b6f0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f3b6f0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-9" prio=10 tid=0x01959210 nid=0x3e in Object.wait() [0xdb1ef000..0xdb1ef970]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f5cfb0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f5cfb0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-8" prio=10 tid=0x031987a8 nid=0x3d in Object.wait() [0xdb23f000..0xdb23fbf0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f5ce78> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f5ce78> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-7" prio=10 tid=0x01bc1278 nid=0x3c in Object.wait() [0xdb28f000..0xdb28fc70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f5cc38> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f5cc38> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-6" prio=10 tid=0x0246f3f8 nid=0x3b in Object.wait() [0xdb2df000..0xdb2dfaf0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f5cb28> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f5cb28> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-5" prio=10 tid=0x026d4bc0 nid=0x3a in Object.wait() [0xdb32f000..0xdb32fb70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f51870> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f51870> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-4" prio=10 tid=0x026d49f8 nid=0x39 in Object.wait() [0xdb37f000..0xdb37f9f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f51150> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f51150> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-3" prio=10 tid=0x024fe0f0 nid=0x38 in Object.wait() [0xdb3cf000..0xdb3cfa70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f50d78> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f50d78> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-2" prio=10 tid=0x01960aa0 nid=0x37 in Object.wait() [0xdb41f000..0xdb41f8f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f50b30> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f50b30> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "DefaultQuartzScheduler_Worker-1" prio=10 tid=0x019608d8 nid=0x36 in Object.wait() [0xdb46f000..0xdb46f970]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x46f3b778> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                     at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
                                     - locked <0x46f3b778> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
                                    
                                    "Thread-8" daemon prio=10 tid=0x01b169a0 nid=0x35 waiting on condition [0xdb64f000..0xdb64fbf0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at org.jboss.console.plugins.AOPLister$1.run(AOPLister.java:882)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=10 tid=0x02174af8 nid=0x33 waiting on condition [0xdbd0f000..0xdbd0faf0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1549)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "WorkerThread#0[64.103.124.247:20580]" prio=10 tid=0x00b1a7c8 nid=0x32 waiting for monitor entry [0xdc02e000..0xdc02fb70]
                                     at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
                                     - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
                                     at com.cisco.nm.cmp.gjc.GWConnectionImpl.execute(GWConnectionImpl.java:225)
                                     at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:820)
                                     at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:802)
                                     at com.cisco.nm.cmp.imomanager.IMOManagerBean.execute(IMOManagerBean.java:639)
                                     at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
                                     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                                     at java.lang.reflect.Method.invoke(Method.java:585)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
                                     at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
                                     at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
                                     at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.aspects.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:166)
                                     at org.jboss.ejb3.security.RoleBasedAuthorizationInterceptor.invoke(RoleBasedAuthorizationInterceptor.java:108)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
                                     at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
                                     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
                                     at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:278)
                                     at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
                                     at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
                                     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
                                     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572)
                                     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373)
                                     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
                                    
                                    "SubscriptionWatchDog" prio=10 tid=0x01780cd8 nid=0x31 waiting on condition [0xdc5ff000..0xdc5ff9f0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "secondaryServerSocketThread" daemon prio=10 tid=0x01e0a468 nid=0x30 runnable [0xdba5f000..0xdba5fa70]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x47104678> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:871)
                                    
                                    "ServerSocketRefresh" daemon prio=10 tid=0x01b22b40 nid=0x2f in Object.wait() [0xdbaaf000..0xdbaaf8f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x471044f0> (a java.lang.Object)
                                     at java.lang.Object.wait(Object.java:474)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                                     - locked <0x471044f0> (a java.lang.Object)
                                    
                                    "AcceptorThread#0:4457" prio=10 tid=0x019e5150 nid=0x2e runnable [0xdbaff000..0xdbaff970]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x471483b8> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "ServerSocketRefresh" daemon prio=10 tid=0x020e2720 nid=0x28 in Object.wait() [0xdc2af000..0xdc2afa70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x47e8e240> (a java.lang.Object)
                                     at java.lang.Object.wait(Object.java:474)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                                     - locked <0x47e8e240> (a java.lang.Object)
                                    
                                    "AcceptorThread#0:3874" prio=10 tid=0x023587a0 nid=0x27 runnable [0xdc2ff000..0xdc2ff8f0]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x47e8e320> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "ServerSocketRefresh" daemon prio=10 tid=0x006ba330 nid=0x25 in Object.wait() [0xdc64f000..0xdc64fbf0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x403fb1b8> (a java.lang.Object)
                                     at java.lang.Object.wait(Object.java:474)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
                                     - locked <0x403fb1b8> (a java.lang.Object)
                                    
                                    "AcceptorThread#0:4446" prio=10 tid=0x00a7a128 nid=0x24 runnable [0xdc69f000..0xdc69fc70]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x4854cd70> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "PooledInvokerAcceptor#0-4445" prio=10 tid=0x016fd678 nid=0x23 runnable [0xdc6ef000..0xdc6efaf0]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x480231f0> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "RMI TCP Accept-4449" daemon prio=10 tid=0x002297a8 nid=0x22 runnable [0xdc73f000..0xdc73fb70]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x47eab170> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "Listener:44855" daemon prio=10 tid=0x01811838 nid=0x21 runnable [0xdc78f000..0xdc78f9f0]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x483b4778> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)
                                    
                                    "Thread-4" daemon prio=10 tid=0x00b30a48 nid=0x1f waiting on condition [0xdc7df000..0xdc7df8f0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)
                                    
                                    "Listener:44852" daemon prio=10 tid=0x0182d968 nid=0x1e runnable [0xdc82f000..0xdc82f970]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x47e86970> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)
                                    
                                    "RMI LeaseChecker" daemon prio=10 tid=0x01371978 nid=0x1d waiting on condition [0xdc87f000..0xdc87fbf0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "JBoss System Threads(1)-2" daemon prio=10 tid=0x00ab9cd8 nid=0x1c runnable [0xdc8cf000..0xdc8cfc70]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x480232d0> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
                                     at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
                                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "GC Daemon" daemon prio=2 tid=0x01508b30 nid=0x1a in Object.wait() [0xdc98f000..0xdc98fb70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x40529548> (a sun.misc.GC$LatencyLock)
                                     at sun.misc.GC$Daemon.run(GC.java:100)
                                     - locked <0x40529548> (a sun.misc.GC$LatencyLock)
                                    
                                    "RMI Reaper" prio=10 tid=0x01508628 nid=0x19 in Object.wait() [0xdc9df000..0xdc9df9f0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x40529720> (a java.lang.ref.ReferenceQueue$Lock)
                                     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
                                     - locked <0x40529720> (a java.lang.ref.ReferenceQueue$Lock)
                                     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
                                     at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "Timer-1" daemon prio=10 tid=0x00a7e248 nid=0x18 in Object.wait() [0xdca2f000..0xdca2fa70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x40316e18> (a java.util.TaskQueue)
                                     at java.lang.Object.wait(Object.java:474)
                                     at java.util.TimerThread.mainLoop(Timer.java:483)
                                     - locked <0x40316e18> (a java.util.TaskQueue)
                                     at java.util.TimerThread.run(Timer.java:462)
                                    
                                    "RMI TCP Accept-1098" daemon prio=10 tid=0x01945548 nid=0x17 runnable [0xdca7f000..0xdca7f8f0]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x403168f0> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "JBoss System Threads(1)-1" daemon prio=10 tid=0x018cb260 nid=0x16 runnable [0xdcc4f000..0xdcc4f970]
                                     at java.net.PlainSocketImpl.socketAccept(Native Method)
                                     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
                                     - locked <0x4044aad8> (a java.net.SocksSocketImpl)
                                     at java.net.ServerSocket.implAccept(ServerSocket.java:450)
                                     at java.net.ServerSocket.accept(ServerSocket.java:421)
                                     at org.jboss.web.WebServer.run(WebServer.java:320)
                                     at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
                                     at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
                                     at java.lang.Thread.run(Thread.java:595)
                                    
                                    "ScannerThread" daemon prio=10 tid=0x01ce6c38 nid=0x15 waiting on condition [0xdcc9f000..0xdcc9fbf0]
                                     at java.lang.Thread.sleep(Native Method)
                                     at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)
                                     at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
                                    
                                    "Timer-0" daemon prio=10 tid=0x014d05d0 nid=0x14 in Object.wait() [0xdccef000..0xdccefc70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x40316cf0> (a java.util.TaskQueue)
                                     at java.util.TimerThread.mainLoop(Timer.java:509)
                                     - locked <0x40316cf0> (a java.util.TaskQueue)
                                     at java.util.TimerThread.run(Timer.java:462)
                                    
                                    "DestroyJavaVM" prio=10 tid=0x000380a0 nid=0x1 waiting on condition [0x00000000..0xffbfd768]
                                    
                                    "Low Memory Detector" daemon prio=10 tid=0x00191e10 nid=0x11 runnable [0x00000000..0x00000000]
                                    
                                    "CompilerThread1" daemon prio=10 tid=0x00190358 nid=0x10 waiting on condition [0x00000000..0xf95feb2c]
                                    
                                    "CompilerThread0" daemon prio=10 tid=0x0018f4e8 nid=0xf waiting on condition [0x00000000..0xf96fe9ac]
                                    
                                    "AdapterThread" daemon prio=10 tid=0x0018e070 nid=0xe waiting on condition [0x00000000..0x00000000]
                                    
                                    "Signal Dispatcher" daemon prio=10 tid=0x0018d268 nid=0xd waiting on condition [0x00000000..0x00000000]
                                    
                                    "Finalizer" daemon prio=10 tid=0x001816d0 nid=0xc in Object.wait() [0xfa2ef000..0xfa2efc70]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x40518a00> (a java.lang.ref.ReferenceQueue$Lock)
                                     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
                                     - locked <0x40518a00> (a java.lang.ref.ReferenceQueue$Lock)
                                     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
                                     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
                                    
                                    "Reference Handler" daemon prio=10 tid=0x0017ff80 nid=0xb in Object.wait() [0xfa3ef000..0xfa3efaf0]
                                     at java.lang.Object.wait(Native Method)
                                     - waiting on <0x400043b8> (a java.lang.ref.Reference$Lock)
                                     at java.lang.Object.wait(Object.java:474)
                                     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
                                     - locked <0x400043b8> (a java.lang.ref.Reference$Lock)
                                    
                                    "VM Thread" prio=10 tid=0x0017dea0 nid=0xa runnable
                                    
                                    "GC task thread#0 (ParallelGC)" prio=10 tid=0x000c9210 nid=0x2 runnable
                                    
                                    "GC task thread#1 (ParallelGC)" prio=10 tid=0x000c9c50 nid=0x3 runnable
                                    
                                    "GC task thread#2 (ParallelGC)" prio=10 tid=0x000ca690 nid=0x4 runnable
                                    
                                    "GC task thread#3 (ParallelGC)" prio=10 tid=0x000cb0d0 nid=0x5 runnable
                                    
                                    "GC task thread#4 (ParallelGC)" prio=10 tid=0x000cbb10 nid=0x6 runnable
                                    
                                    "GC task thread#5 (ParallelGC)" prio=10 tid=0x000cc550 nid=0x7 runnable
                                    
                                    "GC task thread#6 (ParallelGC)" prio=10 tid=0x000ccf90 nid=0x8 runnable
                                    
                                    "GC task thread#7 (ParallelGC)" prio=10 tid=0x000cd9d0 nid=0x9 runnable
                                    
                                    "VM Periodic Task Thread" prio=10 tid=0x00193098 nid=0x12 waiting on condition
                                    


                                    1 2 3 Previous Next