1 Reply Latest reply on Nov 20, 2012 7:02 AM by Nitesh Mehta

    clustering problem with JBoss Messaging in JBoss 5.1

    Nitesh Mehta Newbie

      We have a cluster environment with two nodes (Node1 and Node2) of JBoss 5.1. We have two queues deployed.

      When we bring up the Node1 and view consumer count of queue in admin-console it shows consumer count as 1 for each queue. Till now everything fine and work as expected.

      When we bring up the Node2 of cluster and view consumer count of queues in admin-console of both the node, we observe a random behavior sometime consumer count is incremented to 2 on Node1 whereas  consumer count stays 0 on Node 2 and some time it is increase to 1 on Node2.

      When we have consumers for queues on both the node i.e.

      Node1 Queue1 consumer count =1

      Node1 Queue2 consumer count =1

       

      Node2 Queue1 consumer count=1

      Node2 Queue2 consumer count=1

      Everything work as expected

      BUT

      When we have a situation like

       

      Node1 Queue1 consumer count =2

      Node1 Queue2 consumer count =1

       

      Node2 Queue1 consumer count=0

      Node2 Queue2 consumer count=1

      Then all the messages which reach Node2 Queue1 are not processed.

      When one of the nodes goes down these failover happens and these messages get processed

      This situation is happening very often.

      We tried to upgrade to JBoss Messaging 1.4.5 and JBossRemoting  2.5.2.SP3   but it didn’t help.

       

      messaging-service.xml
      
      <?xml version="1.0" encoding="UTF-8"?>
      
      
      <!--
           The JBoss Messaging service deployment descriptor.
      
      
           $Id: messaging-service.xml 85945 2009-03-16 19:45:12Z dimitris@jboss.org $
       -->
      
      
      <server>
      
      
         <!-- ServerPeer MBean configuration
              ============================== -->
      
      
         <mbean code="org.jboss.jms.server.ServerPeer"
            name="jboss.messaging:service=ServerPeer"
            xmbean-dd="xmdesc/ServerPeer-xmbean.xml">
      
      
            <!-- The unique id of the server peer - in a cluster each node MUST have a unique value - must be an integer -->
      
      
            <attribute name="ServerPeerID">${jboss.messaging.ServerPeerID:0}</attribute>
            
            <!-- The default JNDI context to use for queues when they are deployed without specifying one --> 
            
            <attribute name="DefaultQueueJNDIContext">/queue</attribute>
            
            <!-- The default JNDI context to use for topics when they are deployed without specifying one --> 
            
            <attribute name="DefaultTopicJNDIContext">/topic</attribute>
      
      
                  <attribute name="PostOffice">jboss.messaging:service=PostOffice</attribute>
                  
            <!-- The default Dead Letter Queue (DLQ) to use for destinations.
                 This can be overridden on a per destinatin basis -->
            
            <attribute name="DefaultDLQ">jboss.messaging.destination:service=Queue,name=DLQ</attribute>
            
            <!-- The default maximum number of times to attempt delivery of a message before sending to the DLQ (if configured).
                 This can be overridden on a per destinatin basis -->
            
            <attribute name="DefaultMaxDeliveryAttempts">10</attribute>
            
            <!-- The default Expiry Queue to use for destinations. This can be overridden on a per destinatin basis -->
            
            <attribute name="DefaultExpiryQueue">jboss.messaging.destination:service=Queue,name=ExpiryQueue</attribute>
            
            <!-- The default redelivery delay to impose. This can be overridden on a per destination basis -->
            
            <attribute name="DefaultRedeliveryDelay">10000</attribute>
            
            <!-- The periodicity of the message counter manager enquiring on queues for statistics -->
            
            <attribute name="MessageCounterSamplePeriod">5000</attribute>
            
            <!-- The maximum amount of time for a client to wait for failover to start on the server side after
                 it has detected failure -->
            
            <attribute name="FailoverStartTimeout">60000</attribute>
            
            <!-- The maximum amount of time for a client to wait for failover to complete on the server side after
                 it has detected failure -->
            
            <attribute name="FailoverCompleteTimeout">300000</attribute>
      
      
            <attribute name="StrictTck">false</attribute>
            
            <!-- The maximum number of days results to maintain in the message counter history -->
            
            <attribute name="DefaultMessageCounterHistoryDayLimit">-1</attribute>
            
            <!-- The name of the connection factory to use for creating connections between nodes to pull messages -->
            
            <attribute name="ClusterPullConnectionFactoryName">jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory</attribute>
            
            <!-- When redistributing messages in the cluster. Do we need to preserve the order of messages received
                  by a particular consumer from a particular producer? -->
                  
            <attribute name="DefaultPreserveOrdering">false</attribute>
            
            <!-- Max. time to hold previously delivered messages back waiting for clients to reconnect after failover -->
            
            <attribute name="RecoverDeliveriesTimeout">300000</attribute>
            
            <!-- Set to true to enable message counters that can be viewed via JMX -->
            
            <attribute name="EnableMessageCounters">false</attribute>
            
            <attribute name="SuckerPassword">${queues.database.password}</attribute>
            
            <!-- The name of the server aspects configuration resource
            <attribute name="ServerAopConfig">aop/jboss-aop-messaging-server.xml</attribute>
            -->
            <!-- The name of the client aspects configuration resource
              <attribute name="ClientAopConfig">aop/jboss-aop-messaging-client.xml</attribute>
            -->
      
      
            <depends optional-attribute-name="PersistenceManager">jboss.messaging:service=PersistenceManager</depends>
            
            <depends optional-attribute-name="JMSUserManager">jboss.messaging:service=JMSUserManager</depends>
            
            <depends>jboss.messaging:service=Connector,transport=bisocket</depends>
            <depends optional-attribute-name="SecurityStore"
                proxy-type="org.jboss.jms.server.SecurityStore">jboss.messaging:service=SecurityStore</depends>
         </mbean>
      
      
      </server>
      
        • 1. Re: clustering problem with JBoss Messaging in JBoss 5.1
          Nitesh Mehta Newbie

          Further update on this:

          Actually messaging on Node2 Queue1 was getting processed for some time probably because we are using ClusterPullConnectionFactory but after some time it’s stopped.

          After investigating this problem I found that we are using ClusteredConnectionFactory with MDB, and this factory is not suitable to use with MDB.

          Looking at following posts and issues

          https://community.jboss.org/wiki/JBMCluster “JBoss Messaging Clustering 1.4.x Overview”

          https://issues.jboss.org/browse/JBMESSAGING-842 “The current clustered ConnectionFactory is not apropriate for use with an MDB container. An MDB creates several connections during its deployment. If using a load-balancing clustered ConnectionFactory, they'll end up on different cluster nodes, which is precisely what the MDB container doesn't need.”

           

          https://issues.jboss.org/browse/JBMESSAGING-843 “Clustered ConnectionFactory with no load balancing, but failover capabilities needed.”

           

          I figured out that setting SupportsLoadBalancing to false for ClusteredConnectionFactory will make things work.

           

          Please let me know if my observations are incorrect.