JBOSS EJB CMT ActiveMQ results in Production Outages !!!

[sai_bansi]

Ours is a Integration platform where external partners send us a SOAP Message to our servce endpoint

Our JAVA J2EE applicaton comprising of EJB's CMT is hosed on JBOSS AS 6 with Active MQ

We process the SOAP Message and persist into PostGress database and in the next step we send the message to Active MQ to be consumed by  internal applications like Portal.

We have found recent production outages in these environment. Here are the observations:

1. When we look at the recent outages, it starts with JAVA  applicaton giving an error while trying to call/transact with Postgres database, or Portal EJB, or WSDK,. There may or may not be an issue with the remote system, but for some reason, JAVA  applicaton is complaining.
2. JAVA  applicaton keeps trying to send new messages as well as re-send failed messages to this remote system (incl. EJB RMI calls). This is not at all optimal. JAVA  applicaton does not have any logic to detect it, and be smart about it.
3. Eventually, JAVA  applicaton/AMQ gets filled with stuck messages. It stops processing as a whole, and we can also get errors like OOM or AMQ-Jboss bad connections.
4. JAVA  applicaton stores all important information about a message (be it data, or status like dangling/processed/success/failed/…) in the Postgres database. The only thing AMQ contains is a message id (aka claim check). If we lose the data in DB, we're in trouble. We do not lose anything important by losing the AMQ data. The focus is to ensure DB persistence. We believe if AMQ is not re-started with clean data, there will be errors in JAVA  applicaton-AMQ connection, and it can contribute to message data loss in the Postgres DB.
5. Messages should start flowing at this time. In some cases, an action may be needed on the remote system (e.g., a restart).

I found in the Inbound flow the xxxRequestHandler.java class has

method processObjects()  which has  following code snippet:

// Persist activity

activity = persistChange(serviceXml, basechange, integration);

// Send Message to Queue

sendMessageToQueue(activity, IJNDIConstants.SM_INBOUND_QUEUE);

Please note neither of these methods have Transaction demarcation defined:

Caller method :  processObjects()

Called methods : persistChange() and sendMessageToQueue()

So I am thinking of following annotations:

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)

protected PracticePartnerIntegrationActivity persistChange

@TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)

protectedvoid sendMessageToQueue(Serializable activity, String queueName)

Please note I couldn't find either declarative or annotate transaction demarcation except persistence.xml describes transaction type as JTA. So i am not sure if persistence and sendMessageTo Queue are in single transaction!!!

I am planning to these transaction annotation. Please let me know your thoughts

Please let us know how we can avoid frequent producion outages

[wdfink]

I'm not sure whether I've the right view of your problem.

As you say [2] it looks like that you process the incomming message twice, maybe because of Tx demarcation.

Could you describe a bit the workflow, maybe a picture help that we can see the involved components and the Tx demarcation.