1 2 Previous Next 17 Replies Latest reply on Jun 12, 2007 7:36 PM by timfox

    JBoss Messaging Locked Up

    rtm333

      Hi,

      Today we had a situation where JBoss Messaging (JBM 1.0.1.SP4 on JBoss AS 4.0.5.GA) ceased to send any messages to its clients and required a server restart.

      This is an excerpt from a stack dump at the time of the problem:

      
      "WorkerThread#25[10.196.0.39:4848]" prio=10 tid=0x01efa520 nid=0x6bc2 in Object.wait() [0x7d0ff000..0x7d0ffc70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at java.lang.Object.wait(Object.java:474)
       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
       - locked <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at org.jboss.messaging.core.local.PointToMultipointRouter.add(PointToMultipointRouter.java:129)
       at org.jboss.messaging.core.local.Topic.add(Topic.java:106)
       at org.jboss.messaging.core.local.CoreSubscription.connect(CoreSubscription.java:102)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:283)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90)
       at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:118)
       at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      
      "WorkerThread#22[10.196.43.33:2668]" prio=10 tid=0x029100d0 nid=0x385b in Object.wait() [0x8338f000..0x8338fbf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at java.lang.Object.wait(Object.java:474)
       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
       - locked <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at org.jboss.messaging.core.local.PointToMultipointRouter.remove(PointToMultipointRouter.java:161)
       at org.jboss.messaging.core.local.Topic.remove(Topic.java:126)
       at org.jboss.messaging.core.local.CoreSubscription.disconnect(CoreSubscription.java:111)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:360)
       - locked <0x9c4bf920> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59)
       at sun.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      
      "WorkerThread#21[10.196.166.169:4603]" prio=10 tid=0x0174a560 nid=0x2716 in Object.wait() [0x8667f000..0x8667fa70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at java.lang.Object.wait(Object.java:474)
       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
       - locked <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at org.jboss.messaging.core.local.PointToMultipointRouter.remove(PointToMultipointRouter.java:161)
       at org.jboss.messaging.core.local.Topic.remove(Topic.java:126)
       at org.jboss.messaging.core.local.CoreSubscription.disconnect(CoreSubscription.java:111)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:360)
       - locked <0xa0ede0f8> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59)
       at sun.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      
      "WorkerThread#9[10.196.167.238:4254]" prio=10 tid=0x008c6d60 nid=0xa16 in Object.wait() [0x87b1f000..0x87b1fa70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at java.lang.Object.wait(Object.java:474)
       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
       - locked <0x984b3e80> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
       at org.jboss.messaging.core.local.PointToMultipointRouter.remove(PointToMultipointRouter.java:161)
       at org.jboss.messaging.core.local.Topic.remove(Topic.java:126)
       at org.jboss.messaging.core.local.CoreSubscription.disconnect(CoreSubscription.java:111)
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:360)
       - locked <0xa0e40190> (a java.lang.Object)
       at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$close$aop(ConsumerAdvised.java:59)
       at sun.reflect.GeneratedMethodAccessor159.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      
      "Thread-197" prio=10 tid=0x01eba118 nid=0x46b waiting for monitor entry [0x8504f000..0x8504fbf0]
       at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:219)
       - waiting to lock <0xa0e40190> (a java.lang.Object)
       at org.jboss.messaging.core.local.SingleDestinationRouter.handle(SingleDestinationRouter.java:80)
       at org.jboss.messaging.core.ChannelSupport.push(ChannelSupport.java:1642)
       at org.jboss.messaging.core.ChannelSupport.deliverInternal(ChannelSupport.java:682)
       at org.jboss.messaging.core.ChannelSupport.handleInternal(ChannelSupport.java:843)
       at org.jboss.messaging.core.ChannelSupport$HandleRunnable.run(ChannelSupport.java:1732)
       at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
       at java.lang.Thread.run(Thread.java:595)
      
      "ClientPublishQ-4" prio=10 tid=0x01218828 nid=0xcf in Object.wait() [0x896ee000..0x896ef8f0]
       at java.lang.Object.wait(Native Method)
       at java.lang.Object.wait(Object.java:474)
       at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
       - locked <0x984b3e70> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
       at org.jboss.messaging.core.local.PointToMultipointRouter.handle(PointToMultipointRouter.java:81)
       at org.jboss.messaging.core.local.Topic.handle(Topic.java:86)
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:707)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:449)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:83)
       at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:152)
       at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1414)
       at org.jboss.remoting.Client.invoke(Client.java:511)
       at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
       at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:175)
       at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect17.invoke(TransactionAspect17.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
       at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
       at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
       at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:138)
       at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:190)
      
      


      The full stack dump is some 6000 lines long. If required I will happily provide it by mail.

      This is an absolute show stopper for us!

      Any ideas appreciated.



        • 1. Re: JBoss Messaging Locked Up
          timfox

          This looks like this bug:

          http://jira.jboss.com/jira/browse/JBMESSAGING-836

          Which is fixed in SP5, out very soon... (I am putting the release together now).

          • 2. Re: JBoss Messaging Locked Up
            rtm333

            Hi Tim,

            Unfortunately it seems that the deadlock issue is not fixed in SP5. I have a test scenario where a single publisher and several subscribers are connecting against an out-of-the-box Messaging 1.0.1.SP5 on JBoss AS 4.0.5.GA. Some of the subscribers are closing and creating their JMS session every few seconds to provoke the deadlock. After some time (say, a couple of hours) all JMS activity is blocked.

            I have included below the (in my opinion relevant parts of the) stack traces of the server, the publisher, and two subscribers (blocked in different calls). The stack dumps of the remaining subscribers are identical to the included ones.

            Is there any hope, that this issue might still be fixed in the 1.0.1 branch? Or can you propose any workaround?


            Server warning and stack dump:

            15:23:21,726 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client a5g5f2q-4309mb-f1kmuxga-1-f1kmuykz-4. It is possible the client has exited without closing its connection(s) or there is a network problem. All connection resources corresponding to that client process will now be removed.
            
            "WorkerThread#23[10.196.126.100:55021]" prio=10 tid=0x00b9ca58 nid=0x110 in Object.wait() [0xc2e7f000..0xc2e7fc70]
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:474)
             at org.jboss.messaging.util.Future.getResult(Future.java:51)
             - locked <0xdb7404a8> (a org.jboss.messaging.util.Future)
             at org.jboss.messaging.core.ChannelSupport.handle(ChannelSupport.java:223)
             at org.jboss.messaging.core.local.CoreSubscription.handle(CoreSubscription.java:93)
             at org.jboss.messaging.core.local.PointToMultipointRouter.handle(PointToMultipointRouter.java:97)
             at org.jboss.messaging.core.local.Topic.handle(Topic.java:86)
             at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:705)
             at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:456)
             at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:83)
             at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
             at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:152)
             at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
             at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
             at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723)
             at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101)
             at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
             - locked <0xd5032938> (a java.lang.Class)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:556)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
            "WorkerThread#22[10.199.134.128:39384]" prio=10 tid=0x007cb418 nid=0x103 waiting for monitor entry [0xc397f000..0xc397f8f0]
             at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:101)
             - waiting to lock <0xd5032938> (a java.lang.Class)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:556)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
            "WorkerThread#20[10.199.134.128:39383]" prio=10 tid=0x01cdc860 nid=0xeb runnable [0xc507f000..0xc507f8f0]
             at java.net.SocketInputStream.socketRead0(Native Method)
             at java.net.SocketInputStream.read(SocketInputStream.java:129)
             at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             - locked <0xd9e78100> (a java.io.BufferedInputStream)
             at java.io.FilterInputStream.read(FilterInputStream.java:66)
             at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:806)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:502)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
            "Thread-121" prio=10 tid=0x00cf2c88 nid=0xad waiting for monitor entry [0xc8a7f000..0xc8a7f9f0]
             at org.jboss.jms.server.endpoint.ServerConsumerEndpoint$Deliverer.run(ServerConsumerEndpoint.java:760)
             - waiting to lock <0xda9b8cc0> (a java.lang.Object)
             at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
             at java.lang.Thread.run(Thread.java:595)
            
            "WorkerThread#2[10.199.134.128:39364]" prio=10 tid=0x006bade0 nid=0x3b waiting for monitor entry [0xcf17f000..0xcf17f8f0]
             at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:101)
             - waiting to lock <0xd5032938> (a java.lang.Class)
             at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
             at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:556)
             at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
            "Thread-15" prio=10 tid=0x00536948 nid=0x36 waiting for monitor entry [0xcf67f000..0xcf67fb70]
             at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:219)
             - waiting to lock <0xda9b8cc0> (a java.lang.Object)
             at org.jboss.messaging.core.local.SingleDestinationRouter.handle(SingleDestinationRouter.java:80)
             at org.jboss.messaging.core.ChannelSupport.push(ChannelSupport.java:1642)
             at org.jboss.messaging.core.ChannelSupport.deliverInternal(ChannelSupport.java:682)
             at org.jboss.messaging.core.ChannelSupport.handleInternal(ChannelSupport.java:843)
             at org.jboss.messaging.core.ChannelSupport$HandleRunnable.run(ChannelSupport.java:1732)
             at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
             at java.lang.Thread.run(Thread.java:595)
            
            "Timer-3" daemon prio=10 tid=0x01f12e20 nid=0x35 in Object.wait() [0xcf77e000..0xcf77f9f0]
             at java.lang.Object.wait(Native Method)
             - waiting on <0xd99d00b0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
             at java.lang.Object.wait(Object.java:474)
             at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
             - locked <0xd99d00b0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
             at org.jboss.messaging.core.local.PointToMultipointRouter.remove(PointToMultipointRouter.java:161)
             at org.jboss.messaging.core.local.Topic.remove(Topic.java:126)
             at org.jboss.messaging.core.local.CoreSubscription.disconnect(CoreSubscription.java:111)
             at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.close(ServerConsumerEndpoint.java:360)
             - locked <0xda9b8cc0> (a java.lang.Object)
             at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.remove(ServerConsumerEndpoint.java:555)
             - locked <0xda9b8cc0> (a java.lang.Object)
             at org.jboss.jms.server.endpoint.ServerSessionEndpoint.close(ServerSessionEndpoint.java:426)
             - locked <0xda9b8948> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
             at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:312)
             at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(SimpleConnectionManager.java:199)
             - locked <0xd8e8e880> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)
             at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(SimpleConnectionManager.java:233)
             at org.jboss.remoting.ConnectionNotifier.connectionLost(ConnectionNotifier.java:45)
             at org.jboss.remoting.Lease.notifyClientLost(Lease.java:211)
             at org.jboss.remoting.Lease.access$300(Lease.java:39)
             at org.jboss.remoting.Lease$LeaseTimerTask.run(Lease.java:248)
             at java.util.TimerThread.mainLoop(Timer.java:512)
             at java.util.TimerThread.run(Timer.java:462)
            




            Publisher stack dump:

             [java] "ServerSocketRefresh" daemon prio=1 tid=0x086d4b98 nid=0x3dc6 in Object.wait() [0xaa8ef000..0xaa8ef130]
             [java] at java.lang.Object.wait(Native Method)
             [java] - waiting on <0xb05c2ed0> (a java.lang.Object)
             [java] at java.lang.Object.wait(Object.java:474)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
             [java] - locked <0xb05c2ed0> (a java.lang.Object)
            
             [java] "AcceptorThread#0:3864" prio=1 tid=0xaa6f8900 nid=0x3dc5 runnable [0xaa970000..0xaa9701b0]
             [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
             [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             [java] - locked <0xb05c2f40> (a java.net.SocksSocketImpl)
             [java] at java.net.ServerSocket.implAccept(ServerSocket.java:450)
             [java] at java.net.ServerSocket.accept(ServerSocket.java:421)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
             [java] at java.lang.Thread.run(Thread.java:595)
            
             [java] "main" prio=1 tid=0x0805cb10 nid=0x3dac runnable [0xfeffc000..0xfeffd3e8]
             [java] at java.net.SocketInputStream.socketRead0(Native Method)
             [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
             [java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             [java] at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             [java] - locked <0xb05b6b18> (a java.io.BufferedInputStream)
             [java] at java.io.FilterInputStream.read(FilterInputStream.java:66)
             [java] at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:983)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
             [java] at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
             [java] at org.jboss.remoting.Client.invoke(Client.java:1550)
             [java] at org.jboss.remoting.Client.invoke(Client.java:530)
             [java] at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
             [java] at org.jboss.jms.client.container.TransactionAspect.handleSend(TransactionAspect.java:176)
             [java] at org.jboss.aop.advice.org.jboss.jms.client.container.TransactionAspect16.invoke(TransactionAspect16.java)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
             [java] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
             [java] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
             [java] at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
             [java] at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$send_N3028277934545793941.invokeNext(ClientSessionDelegate$send_N3028277934545793941.java)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
             [java] at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:253)
             [java] at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
             [java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             [java] at java.lang.reflect.Method.invoke(Method.java:585)
             [java] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130)
             [java] at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
             [java] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
             [java] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
             [java] at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
             [java] at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
             [java] at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
             [java] at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
             [java] at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
             [java] at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
             [java] at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:172)
             [java] at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:220)
             [java] at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:147)
             [java] at org.jboss.jms.client.JBossMessageProducer.publish(JBossMessageProducer.java:201)
             [java] at testjms.TestTopicPublisher.sendMessage(TestTopicPublisher.java:40)
             [java] - locked <0xb05bb4b8> (a org.jboss.jms.message.ObjectMessageProxy)
             [java] at testjms.TestTopicPublisher.publishMessages(TestTopicPublisher.java:53)
             [java] at testjms.TestTopicPublisher.main(TestTopicPublisher.java:69)
            





            Subscriber 1 stack dump:

             [java] "Thread-5" prio=10 tid=0x009f22f8 nid=0x1c runnable [0xb337e000..0xb337f8f0]
             [java] at java.net.SocketInputStream.socketRead0(Native Method)
             [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
             [java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             [java] at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             [java] - locked <0xb861a8e0> (a java.io.BufferedInputStream)
             [java] at java.io.FilterInputStream.read(FilterInputStream.java:66)
             [java] at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:983)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
             [java] at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
             [java] at org.jboss.remoting.Client.invoke(Client.java:1550)
             [java] at org.jboss.remoting.Client.invoke(Client.java:530)
             [java] at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledge_354868566256058116.invokeNext(ClientSessionDelegate$acknowledge_354868566256058116.java)
             [java] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
             [java] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledge_354868566256058116.invokeNext(ClientSessionDelegate$acknowledge_354868566256058116.java)
             [java] at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledge_354868566256058116.invokeNext(ClientSessionDelegate$acknowledge_354868566256058116.java)
             [java] at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledge_354868566256058116.invokeNext(ClientSessionDelegate$acknowledge_354868566256058116.java)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate.acknowledge(ClientSessionDelegate.java)
             [java] at org.jboss.jms.client.container.SessionAspect.handlePostDeliver(SessionAspect.java:218)
             [java] at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect6.invoke(SessionAspect6.java)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
             [java] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
             [java] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
             [java] at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
             [java] at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
             [java] at org.jboss.jms.client.delegate.ClientSessionDelegate.postDeliver(ClientSessionDelegate.java)
             [java] at org.jboss.jms.client.remoting.MessageCallbackHandler.postDeliver(MessageCallbackHandler.java:162)
             [java] at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:137)
             [java] at org.jboss.jms.client.remoting.MessageCallbackHandler$ListenerRunner.run(MessageCallbackHandler.java:722)
             [java] at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
             [java] at java.lang.Thread.run(Thread.java:595)
            
             [java] "WorkerThread#0[10.199.134.128:39376]" prio=10 tid=0x008fd578 nid=0x1b runnable [0xb327f000..0xb327f970]
             [java] at java.net.SocketInputStream.socketRead0(Native Method)
             [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
             [java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             [java] at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             [java] - locked <0xb861ddc0> (a java.io.BufferedInputStream)
             [java] at java.io.FilterInputStream.read(FilterInputStream.java:66)
             [java] at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             [java] at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:806)
             [java] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:502)
             [java] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             [java] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
             [java] "ServerSocketRefresh" daemon prio=10 tid=0x00bb74b0 nid=0x17 in Object.wait() [0xb2f7f000..0xb2f7fb70]
             [java] at java.lang.Object.wait(Native Method)
             [java] - waiting on <0xb84ae670> (a java.lang.Object)
             [java] at java.lang.Object.wait(Object.java:474)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
             [java] - locked <0xb84ae670> (a java.lang.Object)
            
             [java] "AcceptorThread#0:3501" prio=10 tid=0x00477880 nid=0x16 runnable [0xb307f000..0xb307f9f0]
             [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
             [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             [java] - locked <0xb849fa70> (a java.net.SocksSocketImpl)
             [java] at java.net.ServerSocket.implAccept(ServerSocket.java:450)
             [java] at java.net.ServerSocket.accept(ServerSocket.java:421)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
             [java] at java.lang.Thread.run(Thread.java:595)
            





            Subscriber 2 stack dump:

             [java] "WorkerThread#0[10.199.134.128:39381]" prio=10 tid=0x001cb7b0 nid=0x51 runnable [0xb327f000..0xb327f970]
             [java] at java.net.SocketInputStream.socketRead0(Native Method)
             [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
             [java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             [java] at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             [java] - locked <0xb8646d10> (a java.io.BufferedInputStream)
             [java] at java.io.FilterInputStream.read(FilterInputStream.java:66)
             [java] at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             [java] at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:806)
             [java] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:502)
             [java] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:379)
             [java] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:161)
            
             [java] "ServerSocketRefresh" daemon prio=10 tid=0x010fa4c8 nid=0x17 in Object.wait() [0xb2f7f000..0xb2f7fa70]
             [java] at java.lang.Object.wait(Native Method)
             [java] - waiting on <0xb849db38> (a java.lang.Object)
             [java] at java.lang.Object.wait(Object.java:474)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)
             [java] - locked <0xb849db38> (a java.lang.Object)
            
             [java] "AcceptorThread#0:3098" prio=10 tid=0x00a87c60 nid=0x16 runnable [0xb307f000..0xb307f9f0]
             [java] at java.net.PlainSocketImpl.socketAccept(Native Method)
             [java] at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
             [java] - locked <0xb84a6f58> (a java.net.SocksSocketImpl)
             [java] at java.net.ServerSocket.implAccept(ServerSocket.java:450)
             [java] at java.net.ServerSocket.accept(ServerSocket.java:421)
             [java] at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
             [java] at java.lang.Thread.run(Thread.java:595)
            
             [java] "main" prio=10 tid=0x00036948 nid=0x1 runnable [0xffbfe000..0xffbfede0]
             [java] at java.net.SocketInputStream.socketRead0(Native Method)
             [java] at java.net.SocketInputStream.read(SocketInputStream.java:129)
             [java] at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
             [java] at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
             [java] - locked <0xb8632c98> (a java.io.BufferedInputStream)
             [java] at java.io.FilterInputStream.read(FilterInputStream.java:66)
             [java] at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:983)
             [java] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
             [java] at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
             [java] at org.jboss.remoting.Client.invoke(Client.java:1550)
             [java] at org.jboss.remoting.Client.invoke(Client.java:530)
             [java] at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
             [java] at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
             [java] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
             [java] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
             [java] at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
             [java] at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
             [java] at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
             [java] at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
             [java] at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
             [java] at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java)
             [java] at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:126)
             [java] at testjms.JmsTestDriver.closeSession(JmsTestDriver.java:64)
             [java] at testjms.JmsTestDriver.resetSession(JmsTestDriver.java:75)
             [java] at testjms.TestTopicSubscriber.main(TestTopicSubscriber.java:57)
            





            • 3. Re: JBoss Messaging Locked Up
              timfox

              It looks like we need to apply the fix for http://jira.jboss.org/jira/browse/JBMESSAGING-505 to the PointToMultipointRouter too.

              This should be fairly straightforward, however the 1.0 branch has basically been closed and we're only going to create patches on that branch for support customers who really insist on not upgrading to 1.2.

              Can you send me your test program that reproduces the issue?

              I will make sure it is fixed in 1.2.0.SP2.

              • 4. Re: JBoss Messaging Locked Up
                timfox

                If you don't want to move to 1.2.0.SP2 then you could make the change yourself and rebuild from source.

                I'm asssuming you don't have a support contract.

                • 5. Re: JBoss Messaging Locked Up
                  rtm333

                  Hi Tim,

                  I sent the test program to you by e-mail.

                  I would have preferred to stick with Messaging 1.0.1 for the time being as we do not need clustering and have performed all our tests with this release. But as this seems to be a dead end, we have to reconsider.

                  Has stability and performance of 1.2.0 now reached a level comparable to 1.0.1?

                  On another thread (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=105371) I read about problems with 1.2.0 and the ServiceBindingManager. Has this been resolved? We are dependent on this to work, as we have to run several instances in parallel on the same hardware.

                  Many thanks,
                  Ralf Torsten

                  • 6. Re: JBoss Messaging Locked Up
                    rtm333

                    Hi Tim,

                    The related JIRA entry JBMESSAGING-962 has been marked as non-reproducible. I have to grant that it was very difficult to produce the deadlock. I think I saw it only a couple of times, although the test program ran for more than 24 hours.

                    I have now started my test program (adapted to use the new client libs) against JBM 1.3.0.GA on AS 4.2.0.GA. After about 15 minutes on the first run the server locked up (although it did not yet on later runs). These are the server stack frames that seem most relevant to me. The complete server stack trace together with some client stack traces is nearly 2000 lines long. I will happily provide it upon request.

                    Thanks,
                    Ralf Torsten



                    "Thread-892" prio=10 tid=0x01913678 nid=0x3d8 runnable [0xcc7ff000..0xcc7ff8f0]
                     at java.net.SocketOutputStream.socketWrite0(Native Method)
                     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                     at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                     at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                     - locked <0xdac9c868> (a java.io.BufferedOutputStream)
                     at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                     at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
                     at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
                     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:969)
                     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:557)
                     at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:339)
                     at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                     at org.jboss.remoting.Client.invoke(Client.java:1550)
                     at org.jboss.remoting.Client.invoke(Client.java:530)
                     at org.jboss.remoting.Client.invokeOneway(Client.java:580)
                     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:687)
                     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:564)
                     at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:296)
                     - locked <0xdac691c0> (a org.jboss.remoting.transport.bisocket.BisocketClientInvoker)
                     - locked <0xdac7be88> (a java.lang.Object)
                     at org.jboss.messaging.core.local.RoundRobinPointToPointRouter.handle(RoundRobinPointToPointRouter.java:120)
                     at org.jboss.messaging.core.ChannelSupport.deliverInternal(ChannelSupport.java:608)
                     at org.jboss.messaging.core.ChannelSupport.deliver(ChannelSupport.java:306)
                     - locked <0xdac66818> (a java.lang.Object)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:965)
                     at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
                     at java.lang.Thread.run(Thread.java:595)
                    
                    
                    "WorkerThread#15[10.196.106.28:60231]" prio=10 tid=0x004ca588 nid=0xb6 waiting for monitor entry [0xcccff000..0xcccffbf0]
                     at org.jboss.messaging.core.ChannelSupport.handleInternal(ChannelSupport.java:765)
                     - waiting to lock <0xdac66818> (a java.lang.Object)
                     at org.jboss.messaging.core.ChannelSupport.handle(ChannelSupport.java:157)
                     at org.jboss.messaging.core.local.PagingFilteredQueue.handle(PagingFilteredQueue.java:114)
                     at org.jboss.messaging.core.plugin.postoffice.DefaultPostOffice.route(DefaultPostOffice.java:333)
                     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:699)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:297)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:80)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:152)
                     at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
                     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                     at java.lang.reflect.Method.invoke(Method.java:585)
                     at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                    
                    "Thread-892" prio=10 tid=0x01913678 nid=0x3d8 runnable [0xcc7ff000..0xcc7ff8f0]
                     at java.net.SocketOutputStream.socketWrite0(Native Method)
                     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                     at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                     at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                     - locked <0xdac9c868> (a java.io.BufferedOutputStream)
                     at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                     at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
                     at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
                     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:969)
                     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:557)
                     at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:339)
                     at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
                     at org.jboss.remoting.Client.invoke(Client.java:1550)
                     at org.jboss.remoting.Client.invoke(Client.java:530)
                     at org.jboss.remoting.Client.invokeOneway(Client.java:580)
                     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:687)
                     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:564)
                     at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:296)
                     - locked <0xdac691c0> (a org.jboss.remoting.transport.bisocket.BisocketClientInvoker)
                     - locked <0xdac7be88> (a java.lang.Object)
                     at org.jboss.messaging.core.local.RoundRobinPointToPointRouter.handle(RoundRobinPointToPointRouter.java:120)
                     at org.jboss.messaging.core.ChannelSupport.deliverInternal(ChannelSupport.java:608)
                     at org.jboss.messaging.core.ChannelSupport.deliver(ChannelSupport.java:306)
                     - locked <0xdac66818> (a java.lang.Object)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:965)
                     at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
                     at java.lang.Thread.run(Thread.java:595)
                    
                    
                    "WorkerThread#15[10.196.106.28:60231]" prio=10 tid=0x004ca588 nid=0xb6 waiting for monitor entry [0xcccff000..0xcccffbf0]
                     at org.jboss.messaging.core.ChannelSupport.handleInternal(ChannelSupport.java:765)
                     - waiting to lock <0xdac66818> (a java.lang.Object)
                     at org.jboss.messaging.core.ChannelSupport.handle(ChannelSupport.java:157)
                     at org.jboss.messaging.core.local.PagingFilteredQueue.handle(PagingFilteredQueue.java:114)
                     at org.jboss.messaging.core.plugin.postoffice.DefaultPostOffice.route(DefaultPostOffice.java:333)
                     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:699)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:297)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:80)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:152)
                     at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
                     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                     at java.lang.reflect.Method.invoke(Method.java:585)
                     at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                     at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:85)
                     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:141)
                     - locked <0xd5302b38> (a java.lang.Class)
                     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
                     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
                     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
                     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
                    
                    
                    "WorkerThread#18[10.196.32.30:4358]" prio=10 tid=0x006623b8 nid=0x2f4 waiting for monitor entry [0xcc8ff000..0xcc8ffaf0]
                     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:105)
                     - waiting to lock <0xd5302b38> (a java.lang.Class)
                     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
                     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
                     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
                     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
                    
                    
                    "Timer-3" daemon prio=10 tid=0x015f4090 nid=0x45 waiting for monitor entry [0xce5ff000..0xce5ffc70]
                     at org.jboss.messaging.core.ChannelSupport.remove(ChannelSupport.java:214)
                     - waiting to lock <0xdac66818> (a java.lang.Object)
                     at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:473)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.localClose(ServerSessionEndpoint.java:795)
                     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:376)
                     at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(SimpleConnectionManager.java:192)
                     - locked <0xd9935d08> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)
                     at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(SimpleConnectionManager.java:227)
                     at org.jboss.remoting.ConnectionNotifier.connectionLost(ConnectionNotifier.java:45)
                     at org.jboss.remoting.Lease.notifyClientLost(Lease.java:211)
                     at org.jboss.remoting.Lease.access$300(Lease.java:39)
                     at org.jboss.remoting.Lease$LeaseTimerTask.run(Lease.java:248)
                     at java.util.TimerThread.mainLoop(Timer.java:512)
                     at java.util.TimerThread.run(Timer.java:462)
                    
                    
                    
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_6145266547759487588.invokeNext(SessionAdvised$send_6145266547759487588.java)
                     at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
                     at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:85)
                     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:141)
                     - locked <0xd5302b38> (a java.lang.Class)
                     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
                     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
                     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
                     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
                    
                    
                    "WorkerThread#18[10.196.32.30:4358]" prio=10 tid=0x006623b8 nid=0x2f4 waiting for monitor entry [0xcc8ff000..0xcc8ffaf0]
                     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:105)
                     - waiting to lock <0xd5302b38> (a java.lang.Class)
                     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
                     at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
                     at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
                     at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
                    
                    
                    "Timer-3" daemon prio=10 tid=0x015f4090 nid=0x45 waiting for monitor entry [0xce5ff000..0xce5ffc70]
                     at org.jboss.messaging.core.ChannelSupport.remove(ChannelSupport.java:214)
                     - waiting to lock <0xdac66818> (a java.lang.Object)
                     at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.localClose(ServerConsumerEndpoint.java:473)
                     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.localClose(ServerSessionEndpoint.java:795)
                     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:376)
                     at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(SimpleConnectionManager.java:192)
                     - locked <0xd9935d08> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)
                     at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(SimpleConnectionManager.java:227)
                     at org.jboss.remoting.ConnectionNotifier.connectionLost(ConnectionNotifier.java:45)
                     at org.jboss.remoting.Lease.notifyClientLost(Lease.java:211)
                     at org.jboss.remoting.Lease.access$300(Lease.java:39)
                     at org.jboss.remoting.Lease$LeaseTimerTask.run(Lease.java:248)
                     at java.util.TimerThread.mainLoop(Timer.java:512)
                     at java.util.TimerThread.run(Timer.java:462)
                    




                    • 7. Re: JBoss Messaging Locked Up
                      timfox

                      Yes, please send me the whole stack traces.

                      • 8. Re: JBoss Messaging Locked Up
                        timfox

                        After my latest changes in TRUNK, I have run Ralf's test program since about 9pm last night (12.5 hrs) with 3 producers (1 x 10 ms, 2 x 2ms) and 5 consumers (2 x 2000 , 1 x 700, 1 x 1000, 1 x 1200 ) and it run without any exceptions and did not lock up.

                        Ralf can you double check with the code from TRUNK?

                        Thanks.

                        • 9. Re: JBoss Messaging Locked Up
                          timfox

                          I forgot to say that memory was stable throughout.

                          • 10. Re: JBoss Messaging Locked Up
                            timfox

                             

                            "timfox" wrote:
                            I *forgot* to say that *memory* was stable throughout.


                            No irony intended ;)

                            Obviously my memory is not so stable as my machines.

                            • 11. Re: JBoss Messaging Locked Up
                              rtm333

                              Tim,

                              Thanks for the quick replies. I will try the fix from TRUNK on Monday.

                              In the meantime here are some further observations from my last tests with GA:

                              I suppose part of the memory issues stem from the use of HSQL (for easier reproducability I use the out-of-the-box messaging configuration for the tests). It happens if one of the test subscribers has a very long session restart cycle time and the publishers produce messages faster than they can be consumed. As is to be expected memory usage starts to grow after more than 200.000 messages build up in the subscription.

                              What I also observed when messages started to be persisted is that the JMS system locked up temporarily several times for about 6 to 8 minutes! After that time message publishing and consuming resumed. I suppose some timer task has performed some cleanup operation at that time, although I have no idea what it might be.

                              • 12. Re: JBoss Messaging Locked Up
                                timfox

                                Don't use HSQL.

                                If you are sending a lot of messages to your queues, and they become full and paging starts occurring, then JBM will start paging to the database.

                                If you are using HSQL, then that "database" is in memory! So you're just going to get an OOM sooner or later.

                                Your "pauses" are probably just long gc pauses as the JVM tries to regain memory as it nears the limit.

                                • 13. Re: JBoss Messaging Locked Up
                                  rtm333

                                  We don't use HSQL for the real application. It's just for this quick messaging test.

                                  The pauses happen long before memory is really filled up. I had fine tuned the publishing rate to reach persistence level, but not exhausting memory. In addition, a full GC took about 6 seconds, but the pauses were 6 to 8 minutes!

                                  • 14. Re: JBoss Messaging Locked Up
                                    timfox

                                    Even so, you should never use paging with HSQL

                                    1 2 Previous Next