clustering problem with JBoss Messaging in JBoss 5.1
We have a cluster environment with two nodes (Node1 and Node2) of JBoss 5.1. We have two queues deployed.
When we bring up the Node1 and view consumer count of queue in admin-console it shows consumer count as 1 for each queue. Till now everything fine and work as expected.
When we bring up the Node2 of cluster and view consumer count of queues in admin-console of both the node, we observe a random behavior sometime consumer count is incremented to 2 on Node1 whereas consumer count stays 0 on Node 2 and some time it is increase to 1 on Node2.
When we have consumers for queues on both the node i.e.
Node1 Queue1 consumer count =1
Node1 Queue2 consumer count =1
Node2 Queue1 consumer count=1
Node2 Queue2 consumer count=1
Everything work as expected
BUT
When we have a situation like
Node1 Queue1 consumer count =2
Node1 Queue2 consumer count =1
Node2 Queue1 consumer count=0
Node2 Queue2 consumer count=1
Then all the messages which reach Node2 Queue1 are not processed.
When one of the nodes goes down these failover happens and these messages get processed
This situation is happening very often.
We tried to upgrade to JBoss Messaging 1.4.5 and JBossRemoting 2.5.2.SP3 but it didn’t help.
messaging-service.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
The JBoss Messaging service deployment descriptor.
$Id: messaging-service.xml 85945 2009-03-16 19:45:12Z dimitris@jboss.org $
-->
<server>
<!-- ServerPeer MBean configuration
============================== -->
<mbean code="org.jboss.jms.server.ServerPeer"
name="jboss.messaging:service=ServerPeer"
xmbean-dd="xmdesc/ServerPeer-xmbean.xml">
<!-- The unique id of the server peer - in a cluster each node MUST have a unique value - must be an integer -->
<attribute name="ServerPeerID">${jboss.messaging.ServerPeerID:0}</attribute>
<!-- The default JNDI context to use for queues when they are deployed without specifying one -->
<attribute name="DefaultQueueJNDIContext">/queue</attribute>
<!-- The default JNDI context to use for topics when they are deployed without specifying one -->
<attribute name="DefaultTopicJNDIContext">/topic</attribute>
<attribute name="PostOffice">jboss.messaging:service=PostOffice</attribute>
<!-- The default Dead Letter Queue (DLQ) to use for destinations.
This can be overridden on a per destinatin basis -->
<attribute name="DefaultDLQ">jboss.messaging.destination:service=Queue,name=DLQ</attribute>
<!-- The default maximum number of times to attempt delivery of a message before sending to the DLQ (if configured).
This can be overridden on a per destinatin basis -->
<attribute name="DefaultMaxDeliveryAttempts">10</attribute>
<!-- The default Expiry Queue to use for destinations. This can be overridden on a per destinatin basis -->
<attribute name="DefaultExpiryQueue">jboss.messaging.destination:service=Queue,name=ExpiryQueue</attribute>
<!-- The default redelivery delay to impose. This can be overridden on a per destination basis -->
<attribute name="DefaultRedeliveryDelay">10000</attribute>
<!-- The periodicity of the message counter manager enquiring on queues for statistics -->
<attribute name="MessageCounterSamplePeriod">5000</attribute>
<!-- The maximum amount of time for a client to wait for failover to start on the server side after
it has detected failure -->
<attribute name="FailoverStartTimeout">60000</attribute>
<!-- The maximum amount of time for a client to wait for failover to complete on the server side after
it has detected failure -->
<attribute name="FailoverCompleteTimeout">300000</attribute>
<attribute name="StrictTck">false</attribute>
<!-- The maximum number of days results to maintain in the message counter history -->
<attribute name="DefaultMessageCounterHistoryDayLimit">-1</attribute>
<!-- The name of the connection factory to use for creating connections between nodes to pull messages -->
<attribute name="ClusterPullConnectionFactoryName">jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory</attribute>
<!-- When redistributing messages in the cluster. Do we need to preserve the order of messages received
by a particular consumer from a particular producer? -->
<attribute name="DefaultPreserveOrdering">false</attribute>
<!-- Max. time to hold previously delivered messages back waiting for clients to reconnect after failover -->
<attribute name="RecoverDeliveriesTimeout">300000</attribute>
<!-- Set to true to enable message counters that can be viewed via JMX -->
<attribute name="EnableMessageCounters">false</attribute>
<attribute name="SuckerPassword">${queues.database.password}</attribute>
<!-- The name of the server aspects configuration resource
<attribute name="ServerAopConfig">aop/jboss-aop-messaging-server.xml</attribute>
-->
<!-- The name of the client aspects configuration resource
<attribute name="ClientAopConfig">aop/jboss-aop-messaging-client.xml</attribute>
-->
<depends optional-attribute-name="PersistenceManager">jboss.messaging:service=PersistenceManager</depends>
<depends optional-attribute-name="JMSUserManager">jboss.messaging:service=JMSUserManager</depends>
<depends>jboss.messaging:service=Connector,transport=bisocket</depends>
<depends optional-attribute-name="SecurityStore"
proxy-type="org.jboss.jms.server.SecurityStore">jboss.messaging:service=SecurityStore</depends>
</mbean>
</server>