Wildfly jta transaction fails when datasources ...| JBoss.org Content Archive (Read Only)

15. Re: Wildfly jta transaction fails when datasources have been idling for a while or reused bad connection?

yangju Mar 18, 2014 6:53 PM (in response to jesper.pedersen)

I noticed in the trace log that the jca's connection subject is null. I wonder if this null subject is one of the problems.

16. Re: Wildfly jta transaction fails when datasources have been idling for a while or reused bad connection?

yangju Mar 20, 2014 12:54 AM (in response to jesper.pedersen)

By debugging into the code (org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl) when the error was happening, I found that in the following method, the c1 sometimes has a state "Detroyed". When the ConnectionListener's state is "Destroyed", the exception is thrown, saying cannot enlist resources. Is it possible to patch this method to check for the Destroyed state?

protected void managedConnectionReconnected(ConnectionListener cl) throws ResourceException

{

try

{

if (shouldEnlist(cl.getManagedConnection()))

cl.enlist();

}

catch (Throwable t)

{

if (trace)

log.trace("Could not enlist in transaction on entering meta-aware object! " + cl, t);

throw new ResourceException(bundle.notEnlistInTransactionOnEnteringMetaAwareObject(), t);

}

17. Re: Wildfly jta transaction fails when datasources have been idling for a while or reused bad connection?

jesper.pedersen Mar 20, 2014 8:17 AM (in response to yangju)

No, you can't enlist a connection when it is destroyed.

You need to check your application and your server logs to fix the problems reported.

18. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

yangju Apr 14, 2014 5:27 PM (in response to jesper.pedersen)

No, we did not reuse the destroy connection. This is only happen when the connection is used to do inserting into a table through jpa native sql. And it only happens the very first time, as if at that time the connection was not ready and the jca was too fast and did not wait long enough. After a month, we still repeatedly run into the same issue.

I was waiting for the Wildfly 8.0.1 to come out as I suspect this is probably a bug in 8.0.0.Final's jca or ironjacamar implementation's bug. But so far after a month's wait, 8.0.1 still has not come out.

19. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

ctomc Apr 14, 2014 5:35 PM (in response to yangju)

WildFly 8.1.0.CR1 was released over the weekend. Read more about it here http://lists.jboss.org/pipermail/wildfly-dev/2014-April/002020.html

20. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

tomjenkinson Apr 15, 2014 5:17 AM (in response to yangju)

The first warning in the logs is:

{quote}

2014-03-13 11:26:24,579 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (default task-10) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@50272749[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@2a7cfdcc connection handles=0 lastUse=1394727984436 trackByTx=true pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@73e85ce6 mcp=SemaphoreArrayListManagedConnectionPool@2ccc6573[pool=slave-1] xaResource=XAResourceWrapperImpl@1c2b3b7d[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@2a7cfdcc pad=false overrideRmValue=null productName=MySQL productVersion=5.5.31-30.3-log jndiName=java:jboss/datasources/slave-1] txSync=null]: com.mysql.jdbc.jdbc2.optional.MysqlXAException: XAER_INVAL: Invalid arguments (or unsupported command)

at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.mapXAExceptionFromSQLException(MysqlXAConnection.java:601) [mysql-connector-java-5.1.29.jar:]

at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.dispatchCommand(MysqlXAConnection.java:584) [mysql-connector-java-5.1.29.jar:]

at com.mysql.jdbc.jdbc2.optional.MysqlXAConnection.start(MysqlXAConnection.java:524) [mysql-connector-java-5.1.29.jar:]

at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:259)

at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.start(XAResourceWrapperImpl.java:203)

at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:716) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]

at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:398) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]

at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.enlist(TxConnectionListener.java:843)

at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:344)

at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:523)

at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:768)

at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:516)

at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:142)

at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:139) [hibernate-core-4.3.1.Final.jar:4.3.1.Final]

<SNIP/>

{quote}

This is coming from:

https://github.com/jbosstm/narayana/blob/5.0.0.Final/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/transaction/arjunacore/TransactionImple.java#L716

This is a known restriction of MySQL:

http://dev.mysql.com/doc/refman/5.0/en/xa-restrictions.html

You need to try to prevent the need to XA JOIN the branches as you resource manager does not support it. Clearly this is easier said than done as your application appears to need this and it would work with most resource managers.The root cause is that multiple calls to enlistResource for the same *resource manager* (different datasources) is being made. I can see from your standalone.xml that is easily possible as you have many xa-datasources pointed at the same db.

1. Is it possible to reconfigure your application to only have one xa-datasource per MySQL server, or are you needing the separate schemas for authentication or something?

2. You could try to get isSameRM to return false for the different datasources using some kind of wrapper. That could have unintended consequences but for this particular scenario it would cause the transaction manager to use a different Xid for the other datasource (deadlock potential).

3. Start different MySQL servers for the different datasources, then isSameRM would more than likely return false for the different connections. This is probably the easiest thing to try in dev.

4. Try a different database, Postgres tends to be quite spec compliant, if just for a quick test you could also try using h2.

I don't think there is anything we can do from the transaction manager side of things. As an example, just commenting out: narayana/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/transaction/arjunacore/TransactionImple.java at 5.0.0.Final ·… would not be correct as the second datasource would not be enlisted in the transaction.

21. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

jesper.pedersen Apr 15, 2014 8:57 AM (in response to yangju)

There is an <is-same-rm-override> on the <xa-pool>, but as Tom said, not the best idea... Maybe the MariaDB community would be more willing to help you...

22. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

yangju Apr 15, 2014 11:53 AM (in response to tomjenkinson)

Thanks Tom and Jesper. Thank you very much!

Yes, the <is-same-rm-override> does solved my issue and I don't see this XA transaction warning and error anymore.

I am little worried by what you said about potential side effect of deadlock. I hope you can explain more on that.

I still have one question, why this error did not happen when I use eap 6.0. The mysql jdbc driver and mysql server itself are not changed. So I still suspect that there is a subtle change on wildfly side (could be the libraries it uses).

Thank again and I will let you guys know if there are more issues coming up during my testing.

23. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

tomjenkinson Apr 16, 2014 8:04 AM (in response to yangju)

Hi Richard,

iirc is-same-rm-override = false means that it will appear as a different resource manager to the transaction manager. This means we will create a new transaction branch for it when it is enlisted. That means that a resource manager that is physically one entity will act as two entities meaning that it would be theoretically possible (depending on the statements you execute) to achieve JDBC deadlock. Its not strictly my area of expertise but I think this would deadlock:

xid 1 inserts parent 1 in table 1 with no commit

xid 2 inserts parent 2 in table 1 with no commit

xid 1 inserts child 1 in table 2 for parent 2 FK onto table 1 - has to wait for xid 2 to commit or rollback

xid 2 inserts child 2 in table 2 for parent 1 FK onto table 1 - has to wait for xid 1 to commit or rollback

If they shared the xid (i.e. isSameRm is true) then it would be OK.

Tom

24. Re: Wildfly jta transaction fails when datasources which are used for inserting. But only fails for the first time to be used.

marklittle Apr 17, 2014 10:34 AM (in response to yangju)

Tom's right. If MySQL is needed and JOIN is also needed then raise an issue with them to get this fixed and refer back to this thread.