10 Replies Latest reply on Dec 15, 2005 9:39 PM by jahlborn

xa transaction failure not handled correctly when lots of me

jahlborn Dec 6, 2005 5:28 PM

well, there really seem to be two issues. first, sending lots of messages within an xa transaction fails when lots > ~15000. second, when the transaction does fail, the messages that were sent still get sent, event though the rest of the transaction rolls back. i'm using oracle on the backend, and the question i have here is: has anyone else seen this behavior (and fixed it through some configuration tweak), or is this a legitimate jboss bug (in which case i will file a jira entry)? This is being run on jboss 4.0.1.

the test code that i am using at this point is included below. I am not using CMT because the actual code doesn't run within an EJB (i just stuck it in one for simplicity of testing).

public void runTest(int numMessagesToSend) throws Exception
{
try {
Transaction tx = null;
LOG.info("Beginning transaction");
getTxManager().begin();
tx = getTxManager().getTransaction();
tx.registerSynchronization(new TxDebugger("", tx.toString()));

XASession session = null;
MessageProducer sender = null;
try {

LOG.info("Creating Session");
session = getHAXAConnection().createXASession();
tx.enlistResource(session.getXAResource());
sender = session.createProducer(getWorkQueue());

LOG.info("Sending " + numMessagesToSend + " messages");
for(int i = 0; i < numMessagesToSend; ++i) {

ObjectMessage msg = session.createObjectMessage();
msg.setObject(new TestXAPayload());
sender.send(msg, DeliveryMode.PERSISTENT, 4,
Message.DEFAULT_TIME_TO_LIVE);
}
LOG.info("Done sending messages");

} finally {
JMSUtil.close(sender, session);
}

jdbcSave(numMessagesToSend);

// commit tx
LOG.info("Committing transaction");
getTxManager().commit();
} finally {
drainQueue();
}

LOG.info("Finished");
}

private void jdbcSave(int numMessagesToSend) throws Exception
{
LOG.info("Saving TestXAState (jdbc)");
DataSource ds = (DataSource)new InitialContext().lookup("java:/DrenchTestDS");

java.sql.Connection conn = null;
java.sql.Statement stmt = null;
try {
conn = ds.getConnection();
stmt = conn.createStatement();

stmt.executeUpdate("INSERT INTO TESTXASTATE(ID, NUMMESSAGESSENT) VALUES ('1', '" + numMessagesToSend + "')");
} finally {
DbUtil.close(stmt, conn);
}
}

private void drainQueue() throws Exception
{
LOG.info("Draining messages");
MessageConsumer consumer = null;
Session session = null;
int numDrained = 0;
try {
session = getHAConnection().createSession(false,
Session.AUTO_ACKNOWLEDGE);
consumer = session.createConsumer(getWorkQueue());
Message msg = null;
while((msg = consumer.receive(5000)) != null) {
++numDrained;
}
} finally {
JMSUtil.close(consumer, session);
}
LOG.info("Drained " + numDrained + " messages");
}

The output of this when run is as follows (notice that the transaction claims to have rolledback--and indeed, the separate jdbc update does not happen--but the messages get sent anyway):

2005-12-06 17:15:08,132 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Beginning transaction
2005-12-06 17:15:08,137 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Creating Session
2005-12-06 17:15:08,365 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Sending 32000 messages
2005-12-06 17:15:12,989 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Done sending messages
2005-12-06 17:15:13,009 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Saving TestXAState (jdbc)
2005-12-06 17:15:14,859 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Committing transaction
2005-12-06 17:15:14,860 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] TxDebugger beforeCompletion, id , txStr TransactionImpl:XidImpl[FormatId=257, GlobalId=james.ws.hmsonline.com/13, BranchQual=, localId=13]
2005-12-06 17:18:39,816 WARN [org.jboss.tm.TransactionImpl] XAException: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=james.ws.hmsonline.com/13, BranchQual=, localId=13] errorCode=XAER_NOTA
javax.transaction.xa.XAException
at com.inet.ora.p.prepare(Unknown Source)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.prepare(XAManagedConnection.java:166)
at org.jboss.tm.TransactionImpl$Resource.prepare(TransactionImpl.java:2100)
at org.jboss.tm.TransactionImpl.prepareResources(TransactionImpl.java:1552)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:319)
at org.jboss.tm.TxManager.commit(TxManager.java:200)
at com.hmsonline.drench.testApp.ejb.TestXABean.runTest(TestXABean.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:113)
at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:51)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:280)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:146)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:123)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
at org.jboss.ejb.Container.invoke(Container.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:144)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:595)
2005-12-06 17:18:42,066 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] TxDebugger afterCompletion, id , txStr TransactionImpl:XidImpl[FormatId=257, GlobalId=james.ws.hmsonline.com/13, BranchQual=, localId=13] status: STATUS_ROLLEDBACK
2005-12-06 17:18:42,067 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Draining messages
2005-12-06 17:23:03,638 INFO [com.hmsonline.drench.testApp.ejb.TestXABean] Drained 32000 messages

1. Re: xa transaction failure not handled correctly when lots o

adrian.brock Dec 6, 2005 7:32 PM (in response to jahlborn)

http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossHelp
Redundant question and obviously not read the release notes for later versions.
Actions
2. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 6, 2005 10:55 PM (in response to jahlborn)

i did read the howto on posting questions and i'm not sure where i missed the mark. i don't see how the question is redundant. you were right that i didn't read the release notes for future versions, however i did do extensive googling and other jboss forum searching beforehand to no avail (maybe i picked the wrong terms?). anyway, after going back through the release notes for the 4.0.x versions and looking at all the jms and transaction manager bugs in jira, i think i have half of an answer. i located JBAS-1589 which accounts for why the messages may be commited on rollback, but i see no answers as to why rollback may have started in the first place. did i miss some other issue about transactions failing under load?
Actions
3. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 7, 2005 5:23 PM (in response to jahlborn)

to follow up on this, i managed to get my test code running on 4.0.3SP1, and the rollback problem has indeed been fixed, but there is still the problem of why the rollback occurs in the first place. so, one of the original questions remains: has anyone else seen this behavior while using jboss xa transactions with messaging, possibly also with an oracle database on the backend.
Actions
4. Re: xa transaction failure not handled correctly when lots o

adrian.brock Dec 7, 2005 8:28 PM (in response to jahlborn)

"also with an oracle database on the backend."

Just maybe this isn't a question you will find answered in the JBossMQ forum.
Maybe this FAQ is elsewhere?

"i don't see how the question is redundant"

I despair ...
http://www.jboss.com/wiki/Wiki.jsp?page=FAQJBossJCA

Database:
"I have problems with OracleXA"
Actions
5. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 8, 2005 9:05 AM (in response to jahlborn)

is it possible that i've already read that page and followed the basic instructions to configure oracle for xa? if you read my original email, xa on oracle works fine until the number of messages in the transaction gets large (as in the example code). you are correct that the answer may be oracle related, but i honestly can't point the finger at oracle or jboss at this point. that's why i was asking the question, in case someone has seen this behavior and figured out how to solve it.
Actions
6. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 8, 2005 9:42 AM (in response to jahlborn)

you're probably right that this is no longer a jbossmq problem (as i've now gotten that question answered). which forum would you recommend that i move my question to? there isn't a forum specifically for transaction related issues. the closest might be the ejb forum?
Actions
7. Re: xa transaction failure not handled correctly when lots o

adrian.brock Dec 9, 2005 11:30 PM (in response to jahlborn)

I just read "errorCode=XAER_NOTA"
as in bad transaction id.

The correct forum is somewhere on Oracle.com
Actions
8. Re: xa transaction failure not handled correctly when lots o

adrian.brock Dec 9, 2005 11:38 PM (in response to jahlborn)

"jahlborn" wrote:
you're probably right that this is no longer a jbossmq problem

It never was.
Actions
9. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 10, 2005 1:03 PM (in response to jahlborn)

actually, it *was* partially a jbossmq bug. i'd say that the messages getting committed on rollback was most certainly a jbossmq problem.

i am indeed pursuing the problem from the oracle side of things. apparently you have some crystal ball and can tell that the jboss transaction manager is communicating correctly with oracle and the only possible cause of the problem is oracle! i, unfortunately, can't verify that jboss is not at fault, so i'll continue to pursue the problem from both ends until i get a fix.
Actions
10. Re: xa transaction failure not handled correctly when lots o

jahlborn Dec 15, 2005 9:39 PM (in response to jahlborn)

just wanted to close out this thread if anyone else happens upon it. i finally found the cause of my spontaneous rollbacks, and it was a jboss issue. it turns out that the XAResource transaction timeouts are controlled independently of the TransactionManager transaction timeouts. in the 4.0.3 version of jboss, a config param was added to the datasource config files "xa-resource-timeout", which enables one to set the XAResource transaction timeout (in seconds). by sufficiently lengthening that timeout (default is 5 minutes), the oracle datasource no longer rollsback mid-commit.
Actions

Go to original post