We are using Jboss 5.1.0 GA cluster which uses Jboss Messaging (1.4.3 GA default version shipped with Jboss 5.1.0 GA). We have a situation after few restarts of the cluster nodes only one node picks up the messages from the Queue.
- We have 2 nodes cluster let us say Node 1(N1) and Node 2(N2).
- We have clustered queues deployed on both the nodes N1 and N2.
- We use clustered XA connection factory to post message in the Queue.
- We have MDBs which pick up the messages from the Queue.
Our product engine puts the messages in the local queue and Office load balances the messages to the cluster node and failover also work fine.
After sometime when we restart a cluster nodes one by one then we get into a state when only one node of the cluster(MDB) picks up the message from the Queue. But we verified that during restart there were no exceptions and errors in the server logs.
After googling over web we found this
and which suggests that re-starting the MDB container brings back the MDBs live and they start picking up the messages
The problems which we are facing with the proposed solution
- This solution does not work for us because we cannot monitor and wait for this state manually and re-start the MDB container. Is there a programmatic way of doing this? Meaning an intelligent way to decide that the system is in such kind of state and we have to restart the MDB containers.
The bigger problem is we really do not understand what is the reason for this? What is making the cluster to in this state.
Please suggest some clean solution.