Ours is a Integration platform where external partners send us a SOAP Message to our servce endpoint
Our JAVA J2EE applicaton comprising of EJB's CMT is hosed on JBOSS AS 6 with Active MQ
We process the SOAP Message and persist into PostGress database and in the next step we send the message to Active MQ to be consumed by internal applications like Portal.
We have found recent production outages in these environment. Here are the observations:
- When we look at the recent outages, it starts with JAVA applicaton giving an error while trying to call/transact with Postgres database, or Portal EJB, or WSDK,. There may or may not be an issue with the remote system, but for some reason, JAVA applicaton is complaining.
- JAVA applicaton keeps trying to send new messages as well as re-send failed messages to this remote system (incl. EJB RMI calls). This is not at all optimal. JAVA applicaton does not have any logic to detect it, and be smart about it.
- Eventually, JAVA applicaton/AMQ gets filled with stuck messages. It stops processing as a whole, and we can also get errors like OOM or AMQ-Jboss bad connections.
- JAVA applicaton stores all important information about a message (be it data, or status like dangling/processed/success/failed/…) in the Postgres database. The only thing AMQ contains is a message id (aka claim check). If we lose the data in DB, we're in trouble. We do not lose anything important by losing the AMQ data. The focus is to ensure DB persistence. We believe if AMQ is not re-started with clean data, there will be errors in JAVA applicaton-AMQ connection, and it can contribute to message data loss in the Postgres DB.
- Messages should start flowing at this time. In some cases, an action may be needed on the remote system (e.g., a restart).
I found in the Inbound flow the xxxRequestHandler.java class has
method processObjects() which has following code snippet:
// Persist activity
activity = persistChange(serviceXml, basechange, integration);
// Send Message to Queue
Please note neither of these methods have Transaction demarcation defined:
Caller method : processObjects()
Called methods : persistChange() and sendMessageToQueue()
So I am thinking of following annotations:
protected PracticePartnerIntegrationActivity persistChange
protectedvoid sendMessageToQueue(Serializable activity, String queueName)
Please note I couldn't find either declarative or annotate transaction demarcation except persistence.xml describes transaction type as JTA. So i am not sure if persistence and sendMessageTo Queue are in single transaction!!!
I am planning to these transaction annotation. Please let me know your thoughts
Please let us know how we can avoid frequent producion outages