TX Recovery in JMS Bridge: Recovery Logs remain on server after recovery

[buehlmann]

Dear community,

I've some troubles with TX-recovery in a JMS bridge. The setup looks like this:

- Server 1: Standalone HornetQ Server runs on JBoss EAP 6.3

- Server 2: Standalone JBoss A-MQ 6.1 Server

- Server 3: JBoss EAP 6.3, with a deployed JMS Bridge. The bridge is transactional and implemented with Apache Camel. It simple routes messages written in a Queue in A-MQ to another Queue in HornetQ

The bridge uses for HornetQ and A-MQ XA-enabled ConnectionFactories. JTA is enabled as well. I use Byteman and the Script provided by http://www.jboss.org/quickstarts/eap/jta-crash-rec to let the VM with the bridge crash in the second phase of the 2PC. This step works well and a recovery log is written to the store.

After that I restarted JBoss EAP (with Byteman disabled) to see if the TX Recovery works. Two minutes after start of JBoss the message will be delivered finally to HornetQ.

BUT: The transaction Logs remain on the server, and every two minutes the Periodic Recovery Thread tries to recover the transaction. I assume that the TX-logs should be deleted after a successfuly recovery. Right?

Attached you will find the relevant snippets of the log files:

bridge-log-during-jvm-crash.log: Log file during the Byteman script lets the VM crash when a message was sent over the JMS bridge.

bridge-log-after-eap-restart.log: The JBoss Log file after the Server was restarted. This might be the interesting part

management-console-transaction-logs.png: Information about the TX-log from the Management console. Screenshot was taken after the message was delivered sucessfully.

jboss-cli-output.log: Information about the TX-log from the JBoss CLI

Any hints would greatly be appreciated!

Ben

[buehlmann]

Anyone any idea?

I've found out that the behaviour is almost the same as in the "jta-crash-rec: Example of JTA Crash Recovery". The transaction logs will not be deleted after a successfully TX recovery and both resource are in the State PREPARED. Is that the expected behaviour?

 

[mnovak]

Hi Benjamin,

I've checked the log before the crash and you're killing server 3 with JMS bridge just when TM sends commit to HornetQ resource manager (ActiveMQ sub-transaction is already commited). Server 3 simply did not received response from HornetQ that commit was successful (because server 3 was killed by byteman) and did not have a chance to update XA transaction in TM journal. But HornetQ committed this transaction in HornetQ journal and because it's committed HornetQ also removed it from HornetQ journal. So he's not aware of this transaction anymore (the same holds for JBoss A-MQ).

What you see after restart of server 3 is - TM finds unfinished transaction in its journal asks JBoss A-MQ and HornetQ to commit it because both of sub-transactions were in prepared state. But this commit already happened before the crash and neither of them holds information of it any longer. So they answer TM - sorry I dont' know this transaction and TM logs warnings about it. There is a timeout (~hours) when TM gives up and removes this journal record.

Thanks,

Mirek

Edit: Warnings are annoying but there is no data integrity problem. All messages were delivered to HornetQ.