0 Replies Latest reply on Mar 2, 2010 1:13 PM by Sudarshan Sharma

    JMS Message Queueing bottleneck

    Sudarshan Sharma Newbie

      We are using jboss-4.0.2. I have around 1500 queues into my application. There are seperate threads for each queues. All thread dumps have the usual pattern that most of threads are waiting for the following lock.  Even the threads that work on seperate Queues have to wait for the same lock. Is there anyway, I can prevent this locking/bottleneck ? Atleast it will be great if  threads working on different queues don't have to wait for the same lock. Application will then be able to scale well.

       

       

      "MessageProcessor1" prio=5 tid=0x03ea7a38 nid=0xee4 waiting for monitor entry [4fdde000..4fddfc30]
          at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getPool(JBossManagedConnectionPool.java:660)
          - waiting to lock <0xb5c6d410> (a java.util.HashMap)
          at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:514)
          at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
          at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
          at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
          at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1368)
          at org.jboss.mq.pm.jdbc2.PersistenceManager.loadFromStorage(PersistenceManager.java:1176)
          at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:402)
          at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:350)
          - locked <0xd8cc50c8> (a org.jboss.mq.server.MessageReference)
          at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:155)
          - locked <0xd8cc50c8> (a org.jboss.mq.server.MessageReference)
          at org.jboss.mq.server.MessageReference.getHeaders(MessageReference.java:248)
          at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:508)
          - locked <0xc239efd0> (a java.lang.Object)
          - locked <0xc239f000> (a org.jboss.mq.server.ReceiversImpl)
          at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:136)
          at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:222)
          at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:656)
          at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:226)
          at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:100)
          at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:570)
          at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:226)
          at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:244)
          at org.jboss.mq.Connection.receive(Connection.java:909)
          at org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:398)
          - locked <0xd8d43620> (a java.util.LinkedList)

         .........

         .........

       

       

      Following is the stack trace of the thread holding the lock. This thread is trying to remove the message from the Queue.

       


      "MessageUser520" prio=5 tid=0x03eaa9f8 nid=0xee8 runnable  [4fc9e000..4fc9fc30]
      at  org.jboss.resource.connectionmanager.JBossManagedConnectionPool$SubjectActions.hashCode(JBossManagedConnectionPool.java:918)
      at  org.jboss.resource.connectionmanager.JBossManagedConnectionPool$SubjectKey.hashCode(JBossManagedConnectionPool.java:803)
      at  java.util.HashMap.hash(HashMap.java:261)
      at  java.util.HashMap.get(HashMap.java:317)
      at  org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getPool(JBossManagedConnectionPool.java:662)
      -  locked <0xb5c6d410> (a java.util.HashMap)
      at  org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:514)
      at  org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:395)
      at  org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:297)
      at  org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:447)
      at  org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      at  org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      at  org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1368)
      at  org.jboss.mq.pm.jdbc2.PersistenceManager.remove(PersistenceManager.java:1025)
      at  org.jboss.mq.server.BasicQueue.acknowledge(BasicQueue.java:573)
      at  org.jboss.mq.server.JMSQueue.acknowledge(JMSQueue.java:123)
      at  org.jboss.mq.server.ClientConsumer.acknowledge(ClientConsumer.java:328)
      at  org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:519)
      at  org.jboss.mq.server.JMSDestinationManager.acknowledge(JMSDestinationManager.java:503)
      at  org.jboss.mq.server.JMSServerInterceptorSupport.acknowledge(JMSServerInterceptorSupport.java:198)
      at  org.jboss.mq.server.TracingInterceptor.acknowledge(TracingInterceptor.java:481)
      at  org.jboss.mq.server.JMSServerInvoker.acknowledge(JMSServerInvoker.java:198)
      at  org.jboss.mq.il.jvm.JVMServerIL.acknowledge(JVMServerIL.java:214)
      at  org.jboss.mq.Connection.send(Connection.java:1139)
      at  org.jboss.mq.SpySession.doAcknowledge(SpySession.java:175)
      - locked  <0xd8898e50> (a java.util.ArrayList)
      at  org.jboss.mq.SpyMessage.doAcknowledge(SpyMessage.java:320)
      at  org.jboss.mq.SpyMessageConsumer.preProcessMessage(SpyMessageConsumer.java:829)
      at  org.jboss.mq.SpyMessageConsumer.receive(SpyMessageConsumer.java:401)
      - locked  <0xdc3b81e8> (a java.util.LinkedList)

       

       

       

      Many thanks in Advance. Any input will be greatly appreciated.

       

       

      Regards,

      Sudarshan

      sudarshan.sharma@gmail.com