8 Replies Latest reply on Mar 17, 2009 8:57 AM by gaohoward

    Server Hang due to dead Lock

      Hi,

      We experienced a server hang a couple of days back and analyzed the thread dump. We use JBoss 4.2 GA, JBM 1.4. The sever seems to hang due to the 2 threads below.

      The Thread: Thread-136 : has the WriterpreferenceLock and the MessageSucker Lock and is waiting for the lock to be released on java.lang.Object@1b6b0a1. While the Thread Thread-50 has the lock on java.lang.Object@1b6b0a1 and is waiting for MessageSucker leading to the Dead Lock Situation.

      Could publish the full dump after I confirm that I have the necessary permissions. Any hint on how to avoid this would really be appreciated.

      Ranga

      Thread: Thread-136 : priority:5, demon:true, threadId:44755, threadState:BLOCKED, lockName:java.lang.Object@1b6b0a1
      org.jboss.messaging.core.impl.MessagingQueue.unregisterSucker(MessagingQueue.java:386)
      ---------------------------------------------------------------------------------
      The previous method call is waiting for Lock java.lang.Object@1b6b0a1
      ---------------------------------------------------------------------------------
      org.jboss.messaging.core.impl.clusterconnection.MessageSucker.stop(MessageSucker.java:178)
      ---------------------------------------------------------------------------------
      The previous method call seems to have the the Lock on Message Sucker.
      ---------------------------------------------------------------------------------
      org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.removeSucker(ClusterConnectionManager.java:508)
      org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.notify(ClusterConnectionManager.java:384)
      org.jboss.messaging.core.impl.DefaultClusterNotifier.sendNotification(DefaultClusterNotifier.java:72)
      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingInMemory(MessagingPostOffice.java:2339)
      ---------------------------------------------------------------------------------
      The previous method call seems to have the the Writer Preference Lock
      ---------------------------------------------------------------------------------
      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.cleanDataForNode(MessagingPostOffice.java:2690)
      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.nodesLeft(MessagingPostOffice.java:963)
      org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener$1ViewChangeRunnable.run(GroupMember.java:517)
      java.lang.Thread.run(Thread.java:595)
      
      
      Thread: Thread-50 : priority:5, demon:false, threadId:220, threadState:BLOCKED,
      lockName:org.jboss.messaging.core.impl.clusterconnection.MessageSucker@eb9dfa
      org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:208)
      ---------------------------------------------------------------------------------
      The previous method call is waiting for Lock org.jboss.messaging.core.impl.clusterconnection.MessageSucker@eb9dfa
      ---------------------------------------------------------------------------------
      org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:565)
      org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:515)
      org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:226)
      -----------------------------------------
      The previous method call seems to have the the lock on java.lang.Object@1b6b0a1"
      -----------------------------------------
      org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1598)
      EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89)
      java.lang.Thread.run(Thread.java:595)
      
      


        • 1. Re: Server Hang due to dead Lock
          gaohoward

          hi, can you please indicate which version of jbm 1.4? Also please give some details about your application configuration, for example, how many nodes are you using?

          • 2. Re: Server Hang due to dead Lock

            Hi,

            We use jboss-messaging-1.4.0.SP1. We currently have four nodes. we start the server using the messaging configuration (-c messaging).

            Please find below our messaging-service.xml.

            Best Regards,
            Ranga.

            <?xml version="1.0" encoding="UTF-8"?>
            
            <!--
             The JBoss Messaging service deployment descriptor.
            
             $Id: messaging-service.xml 3204 2007-10-19 12:42:17Z timfox $
             -->
            
            <server>
            
             <!-- ServerPeer MBean configuration
             ============================== -->
            
             <mbean code="org.jboss.jms.server.ServerPeer"
             name="jboss.messaging:service=ServerPeer"
             xmbean-dd="xmdesc/ServerPeer-xmbean.xml">
            
             <!-- The unique id of the server peer - in a cluster each node MUST have a unique value - must be an integer -->
            
             <attribute name="ServerPeerID">134567</attribute>
            
             <!-- The default JNDI context to use for queues when they are deployed without specifying one -->
            
             <attribute name="DefaultQueueJNDIContext">/queue</attribute>
            
             <!-- The default JNDI context to use for topics when they are deployed without specifying one -->
            
             <attribute name="DefaultTopicJNDIContext">/topic</attribute>
            
             <attribute name="PostOffice">jboss.messaging:service=PostOffice</attribute>
            
             <!-- The JAAS security domain to use for JBoss Messaging -->
            
             <attribute name="SecurityDomain">java:/jaas/messaging</attribute>
            
             <!-- The default security configuration to apply to destinations - this can be overridden on a per destination basis -->
            
             <attribute name="DefaultSecurityConfig">
             <security>
             <role name="guest" read="true" write="true" create="true"/>
             </security>
             </attribute>
            
             <!-- The default Dead Letter Queue (DLQ) to use for destinations.
             This can be overridden on a per destinatin basis -->
            
             <attribute name="DefaultDLQ">jboss.messaging.destination:service=Queue,name=DLQ</attribute>
            
             <!-- The default maximum number of times to attempt delivery of a message before sending to the DLQ (if configured).
             This can be overridden on a per destinatin basis -->
            
             <attribute name="DefaultMaxDeliveryAttempts">10</attribute>
            
             <!-- The default Expiry Queue to use for destinations. This can be overridden on a per destinatin basis -->
            
             <attribute name="DefaultExpiryQueue">jboss.messaging.destination:service=Queue,name=ExpiryQueue</attribute>
            
             <!-- The default redelivery delay to impose. This can be overridden on a per destination basis -->
            
             <attribute name="DefaultRedeliveryDelay">0</attribute>
            
             <!-- The periodicity of the message counter manager enquiring on queues for statistics -->
            
             <attribute name="MessageCounterSamplePeriod">5000</attribute>
            
             <!-- The maximum amount of time for a client to wait for failover to start on the server side after
             it has detected failure -->
            
             <attribute name="FailoverStartTimeout">60000</attribute>
            
             <!-- The maximum amount of time for a client to wait for failover to complete on the server side after
             it has detected failure -->
            
             <attribute name="FailoverCompleteTimeout">300000</attribute>
            
             <!-- The maximum number of days results to maintain in the message counter history -->
            
             <attribute name="DefaultMessageCounterHistoryDayLimit">-1</attribute>
            
             <!-- The name of the connection factory to use for creating connections between nodes to pull messages -->
            
             <attribute name="ClusterPullConnectionFactoryName">jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory</attribute>
            
             <!-- Use XA when pulling persistent messages from a remote node to this one. -->
            
             <attribute name="UseXAForMessagePull">false</attribute>
            
             <!-- When redistributing messages in the cluster. Do we need to preserve the order of messages received
             by a particular consumer from a particular producer? -->
            
             <attribute name="DefaultPreserveOrdering">false</attribute>
            
             <!-- Max. time to hold previously delivered messages back waiting for clients to reconnect after failover -->
            
             <attribute name="RecoverDeliveriesTimeout">300000</attribute>
            
             <!-- The password used by the message sucker connections to create connections.
             THIS SHOULD ALWAYS BE CHANGED AT INSTALL TIME TO SECURE SYSTEM -->
             <attribute name="SuckerPassword">tyigtbhj</attribute>
            
            
             <depends optional-attribute-name="PersistenceManager">jboss.messaging:service=PersistenceManager</depends>
            
             <depends optional-attribute-name="JMSUserManager">jboss.messaging:service=JMSUserManager</depends>
            
             <depends>jboss.messaging:service=Connector,transport=bisocket</depends>
            
             </mbean>
            
            </server>







            • 3. Re: Server Hang due to dead Lock
              gaohoward

              I can't see anything wrong in your configure file. You may consider upgrade the messaging, as we have fixed quite a number of jira issues since sp1. Your problem may have already fixed in the later versions if JBM.

              • 4. Re: Server Hang due to dead Lock

                Hi,

                Is there a specific version that you would recommend that we upgrade to.

                • 5. Re: Server Hang due to dead Lock
                  gaohoward

                  I suggest you to use JBM 1.4.2.GA.SP1 if you are using JBOSS 4.x.

                  • 6. Re: Server Hang due to dead Lock

                    Hi,

                    I really appreciate your responce. Can you please help
                    understand why you think this release would solve this issue. After taking a looking at the codebase of 1.4.0 SP1 and 1.4.2 GA SP1 it doesn't look like the problem would be solved. The locks on the (Object and Sucker)seems to be acquired and released in the same manner in both the versions. I am asking this Question to understand deeper as to how this release would fix the situation, so that the effort to upgrade can be substantiated.

                    Again your responce will be greatly appreciated.

                    Best Regards,
                    Ranga.

                    • 7. Re: Server Hang due to dead Lock
                      gaohoward

                      I'm sorry this issue still exists in 1.4.2.GA sp1. I'll try to write a test to verify this and file a jira if necessary. Thanks.

                      • 8. Re: Server Hang due to dead Lock
                        gaohoward