5 Replies Latest reply on Aug 8, 2007 2:25 PM by timfox

    Messaging hangs

    mwelss

      We are using jboss 4.2.1, and Messaging 1.4.0CR1
      We have a lot of servlets that listen to topics or produce into queues and sooner or later the messaging is completely stuck.

      Here is a sample stacktrace where the thread wants to create a new session. This is were most of the threads are blocked.

      I have the complete thread dump, if neccessary.


      2007-08-01 11:46:51
      Full thread dump Java HotSpot(TM) Server VM (1.6.0_02-b05 mixed mode):

      "http-0.0.0.0-8080-30" daemon prio=10 tid=0x08bc0000 nid=0x3a67 waiting for monitor entry [0x859e0000..0x859e1030]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.jboss.jms.server.ServerPeer.getDefaultDLQInstance(ServerPeer.java:1090)
      - waiting to lock <0x9565fb58> (a org.jboss.jms.server.ServerPeer)
      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.<init>(ServerSessionEndpoint.java:209)
      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:242)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$createSessionDelegate$aop(ConnectionAdvised
      .java:75)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_605233
      5267724906805.java)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_605233
      5267724906805.java)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.createSessionDelegate(ConnectionAdvised.java)
      at org.jboss.jms.wireformat.ConnectionCreateSessionDelegateRequest.serverInvoke(ConnectionCreateSessionDelegateRequest.java:87)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
      at org.jboss.remoting.Client.invoke(Client.java:1550)
      at org.jboss.remoting.Client.invoke(Client.java:530)
      at org.jboss.remoting.Client.invoke(Client.java:518)
      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:186)
      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:157)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$createSessionDelegate$aop(ClientConnectionDel
      egate.java:191)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_
      6052335267724906805.java)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateSessionDelegate(StateCreationAspect.java:128)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect2.invoke(StateCreationAspect2.java)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeNext(ClientConnectionDelegate$createSessionDelegate_
      6052335267724906805.java)
      at org.jboss.jms.client.container.ConnectionAspect.handleCreateSessionDelegate(ConnectionAspect.java:163)
      at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)




        • 1. Re: Messaging hangs
          timfox

           

          "mwelss" wrote:


          I have the complete thread dump, if neccessary.
          /quote]

          Yes please.


          • 2. Re: Messaging hangs
            timfox

            Martin-

            I think I can see a deadlock in the stack you provided, can you try something out for me?

            In the method org.jboss.messaging.core.impl.postoffice.MessagingPostOffice::removeBindingInMemory there is the following code:

             if (queues.isEmpty())
             {
             mappings.remove(binding.condition);
             }
            
             // Send a notification
             ClusterNotification notification = new ClusterNotification(ClusterNotification.TYPE_UNBIND, nodeID, queueName);
            
             clusterNotifier.sendNotification(notification);
            
             return binding;
             }
             finally
             {
             lock.writeLock().release();
             }
             }
            


            Can you move the notification send outside the lock so it reads something like:

             if (queues.isEmpty())
             {
             mappings.remove(binding.condition);
             }
             }
             finally
             {
             lock.writeLock().release();
             }
            
             // Send a notification
             ClusterNotification notification = new ClusterNotification(ClusterNotification.TYPE_UNBIND, nodeID, queueName);
            
             clusterNotifier.sendNotification(notification);
            
             return binding;
             }
            


            and rebuild?

            Sorry, but we are a bit snowed under here (understatement of the century) so any help greatly appreciated.



            • 3. Re: Messaging hangs
              mwelss

              Tim,

              YES! that fixes the problem!
              Thank you very much.

              Will the fix make it into 1.4.0.CR2?

              Cheers,

              Martin

              • 4. Re: Messaging hangs
                timfox

                Yep, we'll put it in CR2.

                • 5. Re: Messaging hangs
                  timfox