Our cluster consist of 2 nodes and we are using JBoss 4.2.3 with JBoss Messaging 1.4.2.GA-SP1. Our application consists of a series of EJB3 message driven beans. When we submit a request to beans, they start processing. If we then press CTRL+C on the console on one of the nodes, the PostOffice is immediately switched over to the other node, which promptly re-processed the message. However the node on which we pressed CTRL+C continuous to process for a little while. This leads to two threads, one on each node doing the same, or very similar work which leads to duplication on the persistence tier. We have already added extra code to be able to resume the process and skip already complete items, however since two thread now process the same work, this happens continuously and my message is eventually abandoned to DLQ.
Do you have any suggestions on how I could handle this situation better. Ideally I would like to stop the work on one node and have it continue / resume on the other node, however right now, it fails over to the other node, while continuing to process on the node which is stopping.
This presents a rather interesting problem. I could keep state of the process in the database and set a stopping / stopped flag. However in case of a real failure (power / network etc), my code would not be executed and the flag would not be set. The same goes for locking the records which I am processing in the database.
Any suggestions on this are appreciated.
