Opening connection from a marked rollback tx

[oglueck]

I have a case where the transaction is marked rollback during the course of a complex process. But I still have to read from the DB and do some more processing. During that a new DB connection is opened (for read-only purposes) behind the scenes and thats where the connection manager complains:

Caused by: javax.resource.ResourceException: Transaction is not active: tx=Trans
actionImpl:XidImpl[FormatId=257, GlobalId=sg1317z.corproot.net/1761318, BranchQu
al=, localId=1761318]
 at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedCo
nnection(TxConnectionManager.java:290)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateC
onnection(BaseConnectionManager2.java:379)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2$Connectio
nManagerProxy.allocateConnection(BaseConnectionManager2.java:812)
 at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(Wrapp
erDataSource.java:88)
 ... 188 more

I think the connection manager should allow to use a transaction that is marked rollback because it's perfectly legal to read from the DB even if the tx is going to be rolled back.

[vickyk]

"oglueck" wrote:
But I still have to read from the DB and do some more processing. During that a new DB connection is opened (for read-only purposes) behind the scenes and thats where the connection manager complains:

Are you not able to tackle this at the application level ?

[oglueck]

No, that's really hard. Because the tx can be set rollback-only at many different points in the process. So I would need to basically check at every read operation and then manually suspend the tx - a lot of boilerplate code. That's too horrible. I rather patch the connection manager.

[vickyk]

"oglueck" wrote:
I rather patch the connection manager.

Have you completed the patch or this is yet to be done ?

[oglueck]

To be done, really. I was hoping to get a comment from a JBoss developer. Because I really think it is a bad misbehaviour.

[vickyk]

For your requirement you really don't need Transaction and from the logs I could make out that you are using local-tx-datasource/xa-tx-datasource .
If you configure no-tx-datasource things should work well , can you try this and let me know if it works ?
Another option which comes to my mind is to disable the connection sharing across transaction , this can be done by setting
<track-connection-by-tx> - whether the connection should be "locked" to the transaction, returning it to the pool at the end of the transaction (default true for Local, false for XA)
Read this
http://wiki.jboss.org/wiki/Wiki.jsp?page=ConfigJCACommon

[oglueck]

We heavily rely on XA as multiple resource adapters are involved in a tx. Using local or no-tx connections is absolutely no option. And we already use track-connection-by-tx because Oracle connections are not capable of sharing the same connection with multiple XA transactions.

Actually the thing is, I don't know why a new connection is opened by Hibernate in this case and why it doesn't use an existing one. Because there should certainly be at least one free connection available that has already been used with this tx. It may be a bug in Hibernate, but I am completely unsure.

But what's wrong with modifying the connection manager to allow this? This should perfectly be legal as I read the specs.

[vickyk]

"oglueck" wrote:

Actually the thing is, I don't know why a new connection is opened by Hibernate in this case and why it doesn't use an existing one. Because there should certainly be at least one free connection available that has already been used with this tx. It may be a bug in Hibernate, but I am completely unsure.

This should be checked , if you are enabling track-connection-by-tx then every time you try to get the connection from the Pool the pool should return you the new handle to the same existing ManagedConnection.

But what's wrong with modifying the connection manager to allow this? This should perfectly be legal as I read the specs.

Playing with the CM should be the last option , it is trivial thing to do .

[oglueck]

the Pool the pool should return you the new handle to the same existing ManagedConnection.

Yes, that's what I thought too should happen. In fact it does most of the time. We can run for days without any problem and suddenly it occurs a couple of times in a row. The effect could depend on the load or on network conditions. Maybe the connection somehow doesn't get returned to the pool in time or something. In anycase there happens something that is not transparent to the application.

[vickyk]

"oglueck" wrote:
Maybe the connection somehow doesn't get returned to the pool in time or something.

Subsequent Connections are taken from the TransactionLocal and not from the Pool , here is the code

public ConnectionListener getConnection(Transaction trackByTransaction, Subject subject, ConnectionRequestInfo cri)
 throws ResourceException
 {
 // Determine the pool key for this request
 boolean separateNoTx = false;
 if (noTxSeparatePools)
 separateNoTx = clf.isTransactional();
 Object key = getKey(subject, cri, separateNoTx);
 SubPoolContext subPool = getSubPool(key, subject, cri);

 InternalManagedConnectionPool mcp = subPool.getSubPool();

 // Are we doing track by connection?
 TransactionLocal trackByTx = subPool.getTrackByTx();

 // Simple case
 if (trackByTransaction == null || trackByTx == null)
 {
 ConnectionListener cl = mcp.getConnection(subject, cri);
 if (traceEnabled)
 dump("Got connection from pool " + cl);
 return cl;
 }

 // Track by transaction
 try
 {
 trackByTx.lock(trackByTransaction);
 }
 catch (Throwable t)
 {
 JBossResourceException.rethrowAsResourceException("Unable to get connection from the pool for tx=" + trackByTransaction, t);
 }
 try
 {
 // Already got one
 ConnectionListener cl = (ConnectionListener) trackByTx.get(trackByTransaction);
 if (cl != null)
 {
 if (traceEnabled)
 dump("Previous connection tracked by transaction " + cl + " tx=" + trackByTransaction);
 return cl;
 }
 }
 finally
 {
 trackByTx.unlock(trackByTransaction);
 }

 // Need a new one for this transaction
 // This must be done outside the tx local lock, otherwise
 // the tx timeout won't work and get connection can do a lot of other work
 // with many opportunities for deadlocks.
 // Instead we do a double check after we got the transaction to see
 // whether another thread beat us to the punch.
 ConnectionListener cl = mcp.getConnection(subject, cri);
 if (traceEnabled)
 dump("Got connection from pool tracked by transaction " + cl + " tx=" + trackByTransaction);

 // Relock and check/set status
 try
 {
 trackByTx.lock(trackByTransaction);
 }
 catch (Throwable t)
 {
 mcp.returnConnection(cl, false);
 if (traceEnabled)
 dump("Had to return connection tracked by transaction " + cl + " tx=" + trackByTransaction + " error=" + t.getMessage());
 JBossResourceException.rethrowAsResourceException("Unable to get connection from the pool for tx=" + trackByTransaction, t);
 }
 try
 {
 // Check we weren't racing with another transaction
 ConnectionListener other = (ConnectionListener) trackByTx.get(trackByTransaction);
 if (other != null)
 {
 mcp.returnConnection(cl, false);
 if (traceEnabled)
 dump("Another thread already got a connection tracked by transaction " + other + " tx=" + trackByTransaction);
 return other;
 }

 // This is the connection for this transaction
 cl.setTrackByTx(true);
 trackByTx.set(cl);
 if (traceEnabled)
 dump("Using connection from pool tracked by transaction " + cl + " tx=" + trackByTransaction);
 return cl;
 }
 finally
 {
 trackByTx.unlock(trackByTransaction);
 }
 }

If you enabled track-connection-by-tx and you are getting the connections in applicaition in a transaction then the underlying connection should be same .

You can try this code

InitialContext context = new InitialContext();
 DataSource ds = (DataSource)context.lookup("java:XAOracleDS");
 //UserTransaction utx = (UserTransaction)context.lookup("UserTransaction");
 //utx.begin();
 Connection con = ds.getConnection();
 Connection con1 = ds.getConnection();
 WrappedConnection wrap = (WrappedConnection)con;
 WrappedConnection wrap1 = (WrappedConnection)con1;
 out.println("UnderlyingConnection is :---> "+wrap.getUnderlyingConnection()+"<br>");
 out.println("Underlying Connection "+wrap1.getUnderlyingConnection()+"<br>");
 //utx.commit();
 con.close();
 con1.close();

In the above code the underlying connections would be different because the transaction related code is commented , if you uncomment that part it will give you same underlying connection.

So make sure that the part of code which is giving you different result is in transaction .

[vickyk]

Huh ... We have deviated from the original problem :)

[oglueck]

Yes, that's what I expected (that connections are stored in the tx). Hibernate (EJB3) is the one requesting the connection. I am not doing this explicitly there. I might open a connection from the same DS explicitly in another place (within this tx) and then call close() when done. But that should be fine I guess. Of course all work is done in one big tx. It is opened via a UserTransaction from a BMT session bean. The problem occurs before the call returns to the bean that started the tx.

Maybe there is a bug in an interceptor, so the tx is lost in the TransactionLocale?

Anyway, yes we are drifting away from the problem. I still think it is perfectly legal to open as many connections as I like even if the current tx is set rollback-only. I guess the rollback-only flag is only set in the transaction manager and the connection is completely unaware of this fact until the transaction manager performs the rollback at the end.

[vickyk]

"oglueck" wrote:

Anyway, yes we are drifting away from the problem. I still think it is perfectly legal to open as many connections as I like even if the current tx is set rollback-only. I guess the rollback-only flag is only set in the transaction manager and the connection is completely unaware of this fact until the transaction manager performs the rollback at the end.

When the getConnection() is called on the ConnectionFactory for the xa-tx-datasource the AS should make sure that the associated XA Resource is enlisted in the associated Transaction .
Now if you have the following sequence
1) Start TX T1
2) CF.getConnection(); The associated XAResource with the connection will get enlisted in the T1.
3) SET ROLLBACK on T1 . This step basically sets

case Status.STATUS_PREPARED:
 status = Status.STATUS_MARKED_ROLLBACK;

Look at
http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_0/transaction/src/main/org/jboss/tm/TransactionImpl.java
4) Again calling the CF.getConnection() will be able to get the same ManagedConnection from the TransactionLocal but when the associated XAResource is getting enlisted in associated TX this code will get called

// Inactive transaction
 Transaction threadTx = tm.getTransaction();
 if (threadTx == null || status != Status.STATUS_ACTIVE)
 {
 String error = "Transaction " + threadTx + " is not active " + TxUtils.getStatusAsString(status);
 if (trace)
 log.trace(error + " cl=" + this);
 throw new IllegalStateException(error);
 }

http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_0/connector/src/main/org/jboss/resource/connectionmanager/TxConnectionManager.java

So Connection would not be taken and you would get IllegalStateException .....
Enable to TRACE logging on the org.jboss.resource(JBoss JCA) and org.jboss.tm(Transaction Manager) and figure out what is going on ....

[oglueck]

Thanks. I have already had a look at the TM code and saw that check. Trace logging is no option for me. This happens randomly and rarely on a production system.

[vickyk]

"oglueck" wrote:
This happens randomly and rarely on a production system.

It then becomes difficult to identify the problem , the only option I can think right now is to simulate the test on the development system .
If you are able to do it there with the TRACE logging enabled for JCA and TM it would be easy to SPOT the cause.