Transaction Manager patchy or How to recover successfully fr
mreasy Feb 23, 2009 8:43 AMHi,
We experienced the symptom of JMS-messages being processed successfully by the JMS-provider (and removed from its context, i.e. MessageCache) but due to XA-Errors not being deleted from the database. This leads to the messages being re-read on the next JBoss-start and being treated as new messages which are processed again ==> duplicate message-processing.
See https://jira.jboss.org/jira/browse/JBAS-6498 for the same symptom, but under the assumption that it derived from DB-errors directly.
Stacktrace, for failing XA-transaction (stacktrace actually showing the end-method, whereas the commit-method seems to be the problem):
org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory End transaction failed for XAResource oracle.jdbc.xa.OracleXAException at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:938) at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:385) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.end(XAManagedConnection.java:147) at org.jboss.tm.TransactionImpl$Resource.endResource(TransactionImpl.java:2143) at org.jboss.tm.TransactionImpl$Resource.endResource(TransactionImpl.java:2118) at org.jboss.tm.TransactionImpl.endResources(TransactionImpl.java:1462) at org.jboss.tm.TransactionImpl.beforePrepare(TransactionImpl.java:1116) at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:324) at org.jboss.tm.TxManager.commit(TxManager.java:240) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170) at org.jboss.mq.SpySession.run(SpySession.java:323) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748) at java.lang.Thread.run(Thread.java:619)
Now what I did to provoke this error was to hack exceptions into XAMAnagedConnection in a way it would be possible for the JDBC-driver to throw (will attach a diff towards Branch_4_2).
This hack always shows the problem of JMS-messages laying around in the DB and not available (anymore) in the JMS-provider.
Now Adrian came into play and told us to rtfm and activate XARecovery as described in https://jira.jboss.org/jira/browse/JBAS-6498.
Did so and from a first sight it seemed better (messages actually processed having TXOP 'D' in DB) but on the next restart those messages were not deleted as one would have expected.
The problems were present with any tupel of the following:
JBoss 4.0.5_GA with/without XARecovery
JBoss 4.2.3_GA with/without XARecovery
JBoss Branch_4_2 (upcoming 4.2.4) with/without XARecovery
Oracle 10 / 11
Oracle_JAVA_PACKAGE installed /uninstalled to allow DB-side Java-XA-transactions
Orale rights for sys.dba_pending_transactions, sys.dba_2pc_pending, sys.pending_trans and sys.dbms_system available /unavailable
Oracle-THIN-driver was used (versions 1.4, 5 and 6 make no difference here).
So you see we had arjuna and the old TM under test.
Some combinations with XARecovery activated even showed a lot worse behaviour resulting in messages being multiplied, or DB-records being locked forever with arjuna (but that's worth another topic).
So finally the question:
- Is this a general bug in the transaction manager?
- Are we missing sth. which is not described in any of these
- http://management-platform.blogspot.com/2008/11/transaction-recovery-in-jbossas.html
- http://www.jboss.org/community/docs/DOC-10013
- Usual topics in Jira and forum?
- Is our use-case b.s.? Nevertheless it would be valid, since experienced 'in-the-wild'.
Thanks for any light you can shed on this ;)
Regards
Rico