JBoss 5.1 Cluster - HASingleton MDB does not consume messages in sequence after failover to another node of the Cluster
Hi All,
We have a requirement where the messages of certain queues should always be consumed by a Singleton MDB in a two-node cluster to ensure sequential processing and also have the MDB as a HA Singleton to support failover.
We have achieved this by deploying the MDB in the "deploy-hasingleton" directory of the server "all" and we use "farm" deployment to deploy the Session beans , Queue Service ( The queues are set with the attribute "Clustered" true ) and the ConnectionFactory ( we use a ClusteredConnectionFactory that supports Failover and Round-Robin Load Balancing ). The classes referred by the application are placed in "lib" directory.
This is similar to the info in the discussion http://community.jboss.org/message/527363
For testing purpose, we have added sleep of 100 milli-seconds in the method used by the MDB to ensure that the time taken by the MDB to process is more than the time taken by the Client to send the request to the Session Bean Service and the time taken by Session Bean to produce the message.
This is to verify that the messages are processed in sequence.
We have tested this with a client that sends requests to the Cluster. The requests are load-balanced between the two nodes of the cluster and the MDB consumed the messages in the sequence they were placed in the queue.
We added “log.error” statements in the Stateless Session Bean (SLSB) service and the Message-driven Bean (MDB) service to print the message-detail that was sent by the client to the Stateless Session Bean service and the message currently being processed by the MDB.
The Client program posts 20 message objects to the Cluster, setting an attribute “sessionNo” with the message counter value incremented by 1 during each iteration.
Following was the output when both the nodes of the cluster were active. The HASingleton MDB was available on Node1 of the cluster. We observed that the Messages were consumed in the order they reached the Queue destination.
TEST 1 - HASingleton MDB running in Node 1 and both the nodes of the Cluster being active
Output at Server 1
SLSB sessionNo attribute :1
MDB sessionNo attribute :1
SLSB sessionNo attribute :3
SLSB sessionNo attribute :5
MDB sessionNo attribute :2
SLSB sessionNo attribute :7
MDB sessionNo attribute :3
SLSB sessionNo attribute :9
MDB sessionNo attribute :4
SLSB sessionNo attribute :11
MDB sessionNo attribute :5
SLSB sessionNo attribute :13
MDB sessionNo attribute :6
SLSB sessionNo attribute :15
MDB sessionNo attribute :7
SLSB sessionNo attribute :17
MDB sessionNo attribute :8
SLSB sessionNo attribute :19
MDB sessionNo attribute :9
MDB sessionNo attribute :10
MDB sessionNo attribute :11
MDB sessionNo attribute :12
MDB sessionNo attribute :13
MDB sessionNo attribute :14
MDB sessionNo attribute :15
MDB sessionNo attribute :16
MDB sessionNo attribute :17
MDB sessionNo attribute :18
MDB sessionNo attribute :19
MDB sessionNo attribute :20
Output At Server2
SLSB sessionNo attribute :2
SLSB sessionNo attribute :4
SLSB sessionNo attribute :6
SLSB sessionNo attribute :8
SLSB sessionNo attribute :10
SLSB sessionNo attribute :12
SLSB sessionNo attribute :14
SLSB sessionNo attribute :16
SLSB sessionNo attribute :18
SLSB sessionNo attribute :20
TEST 2 - HASingleton MDB running in Node 1 and Node1 shutdown while processing to failover the HASingletonMDB to Node2 of the Cluster
Now we repeated the same test and we brought down the Node1 of the cluster to check if the service gets started at the Node2 of the cluster and messages are consumed.
Output at Server 1
MDB sessionNo attribute :1
SLSB sessionNo attribute :2
MDB sessionNo attribute :2
SLSB sessionNo attribute :4
MDB sessionNo attribute :3
SLSB sessionNo attribute :6
MDB sessionNo attribute :4
SLSB sessionNo attribute :8
MDB sessionNo attribute :5
SLSB sessionNo attribute :10
MDB sessionNo attribute :6
SLSB sessionNo attribute :12
MDB sessionNo attribute :7
SLSB sessionNo attribute :14
MDB sessionNo attribute :8
The messages with attribute value 10, 12 and 14 produced from Node 1 are yet to be processed. Meanwhile the messages with attribute value 9,11,13, 15 ….20 were produced from the Node2 of the cluster.
Output at Server 2
SLSB sessionNo attribute :1
SLSB sessionNo attribute :3
SLSB sessionNo attribute :5
SLSB sessionNo attribute :7
SLSB sessionNo attribute :9
SLSB sessionNo attribute :11
SLSB sessionNo attribute :13
SLSB sessionNo attribute :15
SLSB sessionNo attribute :16
SLSB sessionNo attribute :17
SLSB sessionNo attribute :18
SLSB sessionNo attribute :19
SLSB sessionNo attribute :20
MDB sessionNo attribute :10
MDB sessionNo attribute :12
MDB sessionNo attribute :14
MDB sessionNo attribute :9
MDB sessionNo attribute :11
MDB sessionNo attribute :13
MDB sessionNo attribute :15
MDB sessionNo attribute :16
MDB sessionNo attribute :17
MDB sessionNo attribute :18
MDB sessionNo attribute :19
MDB sessionNo attribute :20
We expected to see the MDB to process the message with attribute value 9,10,11,12…20
But the sequence that observed was 10,12,14 ( the 3 messages that were created by Node 1 before failure) and then 9,11,13,15,16,17,18,19,20.
We could see that only the messages that were produced by Node1 and the messages produced by Node2 during the time of failure had gone out-of-sequence. We can observe that the message processing is sequential from that point ( 15,16,…20)
TEST 3 - HASingleton MDB running in Node 2 and Node2 shutdown while processing to failover the HASingletonMDB to Node1 of the Cluster
We repeated this test with HA Singleton MDB available on Node 2 and failing over the service to Node1 of the cluster.
Following was the output.
Output at Server 2
MDB sessionNo attribute :1
SLSB sessionNo attribute :2
MDB sessionNo attribute :2
SLSB sessionNo attribute :4
MDB sessionNo attribute :3
SLSB sessionNo attribute :6
MDB sessionNo attribute :4
SLSB sessionNo attribute :8
MDB sessionNo attribute :5
SLSB sessionNo attribute :10
MDB sessionNo attribute :6
SLSB sessionNo attribute :12
MDB sessionNo attribute :7
SLSB sessionNo attribute :14
MDB sessionNo attribute :8
SLSB sessionNo attribute :16
MDB sessionNo attribute :9
SLSB sessionNo attribute :18
The messages with attribute value 10, 12 ,14,16,18 produced at Node 2 are yet to be processed. Meanwhile the messages with attribute value 11,13, 15 ….20 were produced from the Node1 of the cluster.
Output at Server 1
SLSB sessionNo attribute :1
SLSB sessionNo attribute :3
SLSB sessionNo attribute :5
SLSB sessionNo attribute :7
SLSB sessionNo attribute :9
SLSB sessionNo attribute :11
SLSB sessionNo attribute :13
SLSB sessionNo attribute :15
SLSB sessionNo attribute :17
SLSB sessionNo attribute :19
SLSB sessionNo attribute :20
MDB sessionNo attribute :11
MDB sessionNo attribute :13
MDB sessionNo attribute :15
MDB sessionNo attribute :17
MDB sessionNo attribute :19
MDB sessionNo attribute :20
MDB sessionNo attribute :10
MDB sessionNo attribute :12
MDB sessionNo attribute :14
MDB sessionNo attribute :16
MDB sessionNo attribute :18
The sequence that observed was 11,13,15,17,19,20 and then 10,12,14,16,18 ( the messages that were produced by Node 2 before failure)
We could see that in both the case the messages generated by Node1 of the cluster is being consumed first even though the request from the client and the messages were produced by the Session Bean service was sequential.
Could someone help us in finding out what we have missed here.
I apologize for such a long description of the tests.