Is HornetQ local tx recovery single threaded?

[billy.sjoberg]

Hi guys,

JBoss 6.1 Final

HornetQ 2.2.5

JBoss ESB 4.10

I'm testing failure scenarios using HornetQ and JBoss ESB and I have a problem with the recovery of transactional messages.

I am using the jms-xa.rar but I've set it to useLocalTX as Hornet is the only TX participant and I dont want to drag XA in unless necessary. JBoss ESB unfortunately does not play well without the JCA adapter with regards to transactions, so I'm using it even though a local tx would do.

All works fine and dandy during happy cases, but if I force a server shutdown I notice that all recovered messages are handled in one thread. Sending messages straight to the queue I can see that the threading works during normal operation, but as long as I'm picking up recovery messages I'm basically single threaded.

Is there some way I can configure how many threads the tx recovery uses?

In my naive transactional world I believed that the TX recovery should just hand off all messages to the queue for resubmit, but I clearly see that the recovered messages seems to be coupled with the application as I cant browse them with my own home-built HornetQ admin console.

[ataylor]

recovery is controlled by the transaction recovery manager and has nothing to do with HornetQ, basically we provide hooks thatallow the recovery manager access to ther HornetQ Resource Manager. As far as i know tho th erecovery manager  will make a pass every n seconds and a single thread will  handle any transactions that need to be handled by either commiting them or rolling them back.

[jbertram]

I'm a bit confused on a few points:

• I'm not aware of any file named jms-xa.rar.  However, I'm familiar with jms-ra.rar which is a JCA RA supporting both XA and local transactions.
• You say you tried to use local transactions, but that ESB "does not play well without the JCA adapter."  However, (as mentioned in the point above) the JCA RA should support both local and XA transactions.  I'm not terribly familiar with ESB so can you clarify what does and does not work with ESB in regards to JCA and transactions?

A few clarifying points/questions:

• Even in the case where multiple resources are enlisted in the transaction, if the resources all come from the same resource manager the transaction manager should optimize the commit to a single phase since a two-phase-commit is not necessary (since there's only one RM involved).  If that optimization takes place (and it should if it isn't) then there shouldn't be anything to recover since only two-phase-commits are logged by the transaction manager.  Can you be more specific about exactly what messages need to be recovered and how you know they need to be recovered?
• HornetQ doesn't actually recover the messages.  This is done by the JBossTS transaction recovery manager.  To my knowledge, recovery can't be configured to use more than one thread.  HornetQ is just a resource manager.  JBossTS is the transaction manager.

[billy.sjoberg]

You're right Justin, it should be jms-ra.rar. Typo by me..

For the second confused point you can count me in :-)

Short to say, the ESB is a bit complex when it comes to its JMS integration. You basically have two types of "esb listeners" as they call it. One is based on a pure jms connection and does not use a resource adapter while the other one is a generic JCA adapter integration. The pure one can not enlist local transactions from what I've found which forced me to use the JCA adapter one even though I only need local transactions.

However, forget that little rant as that is not relevant to the case. Just view my application as a messagelistener that uses the JCA adapter with "useLocalTx=true" in its activation parameter list.

My application simply receives messages from a single queue and forwards them to another one, just for simple testing.

It is multithreaded and the JCA adapter sets up 15 consumers on the queue. Inside the application I use Thread.sleep(y) just to force the messages to be stuck while I deliberately kill my JBoss. (This is done to test the recovery)

* Before I kill the server i notice that all my messages are delivered in parallel, the Thread.sleep is invoked almost instantly for 15 messages by separate threads.

* After restart the messages that were rolled back (15 of them) are delivered to the application again, but this time one by one. So I can clearly see that instead of taking 1 x y milliseconds it now takes 15 x y milliseconds.

(*) If I send 15 more messages (that was not part of the rollback) these are handled concurrently.

This confuses me as I thought that the transaction manager would actually put the non-commited messages back to the queue, not to my application. If that would've been the case it would hand them over and then my app would continue to fetch them concurrently.

I'm obviously a bit of a newb when it comes to how the transactions are handled in JBoss, but thats just what I assumed would happen.

Hope my question became a bit more clear. As I said, treat my assumptions with a grain of salt as I was simply confused with the way tx handling seemed to work.

[ataylor]

This confuses me as I thought that the transaction manager would actually put the non-commited messages back to the queue, not to my application. If that would've been the case it would hand them over and then my app would continue to fetch them concurrently.

The transaction recovery manager simply commits or rolls back the tx depending on its state, if they were marked for rollback they would be rolled back or commited if all branches were prepared successfully