Are exception-sorter expected to work for xa-datasource?

[smarlow]

Are exception-sorter expected to work for xa-datasource?

The dtd @ http://www.jboss.org/j2ee/dtd/jboss-ds_1_5.dtd seems to think it should work.

Our debuger breakpoints on exception-filter only seem to get hit on non-xa datasources but not xa-datasources.

Thanks,
Scott

[weston.price]

Exception sorters are transaction independent. Post your *-ds.xml file so we can take a look as well as the version of JBoss and DB that you are using.

[nathanmeyers]

I'm working with Scott on this issue. Here are some more details:

We're working in JBoss 4.0.4, JDK5, Microsoft SQL Server 2000, Windows XP Pro SP2, JNetDirect 5.532 drivers.

Here is an excerpt from the datasource declarations:


<xa-datasource>
<jndi-name>DefaultDS</jndi-name>
<track-connection-by-tx>true</track-connection-by-tx>
<isSameRM-override-value>false</isSameRM-override-value>
<xa-datasource-class>com.jnetdirect.jsql.JSQLXADataSource</xa-datasource-class>
<xa-datasource-property name="URL">jdbc:JSQLConnect://nmeyerspc/database=mydb/asciiStringParameters=true</xa-datasource-property>
<security-domain>DefaultDSRealm</security-domain>
<no-tx-separate-pools/>
<max-pool-size>200</max-pool-size>
<exception-sorter-class-name>util.SQLServerExceptionSorter</exception-sorter-class-name>
</xa-datasource>


The problem we're seeing is that the exception sorter is never called or even instantiated. We've seen it work flawlessly with local-tx-datasource, but never with xa-datasource.

-----

We're seeing a couple of patterns when connection errors are encountered. Here's the first one - a stack trace taken at the time the SQLException is thrown, during a call to a finder:


Thread [PooledInvokerThread-127.0.0.1-0] (Suspended (exception com.jnetdirect.jsql.JSQLException))
com.jnetdirect.jsql.JSQLException.makeFromDriverError(com.jnetdirect.jsql.JSQLConnection, com.jnetdirect.jsql.IOBuffer, java.lang.String, java.lang.String, boolean) line: not available
com.jnetdirect.jsql.DBComms.transmit(byte, com.jnetdirect.jsql.IOBuffer, int, int, boolean) line: not available
com.jnetdirect.jsql.JSQLConnection(com.jnetdirect.jsql.IOBuffer).a(byte, int, boolean) line: not available
com.jnetdirect.jsql.JSQLConnection.new(java.lang.String, java.lang.String, boolean) line: not available
com.jnetdirect.jsql.JSQLConnection.sendStartTran(java.lang.String) line: not available
com.jnetdirect.jsql.JSQLXAResource.start(javax.transaction.xa.Xid, int) line: not available
org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(javax.transaction.xa.Xid, int) line: 117
org.jboss.tm.TransactionImpl$Resource.startResource() line: 2063
org.jboss.tm.TransactionImpl.enlistResource(javax.transaction.xa.XAResource) line: 581
org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist() line: 757
org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist() line: 548
org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(org.jboss.resource.connectionmanager.ConnectionListener) line: 323
org.jboss.resource.connectionmanager.TxConnectionManager(org.jboss.resource.connectionmanager.BaseConnectionManager2).reconnectManagedConnection(org.jboss.resource.connectionmanager.ConnectionListener) line: 501
org.jboss.resource.connectionmanager.TxConnectionManager(org.jboss.resource.connectionmanager.BaseConnectionManager2).allocateConnection(javax.resource.spi.ManagedConnectionFactory, javax.resource.spi.ConnectionRequestInfo) line: 382
org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(javax.resource.spi.ManagedConnectionFactory, javax.resource.spi.ConnectionRequestInfo) line: 812
org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection() line: 88
org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery(org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand).execute(java.lang.String, java.lang.Object[], int, int, org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCEntityBridge, org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCCMPFieldBridge, org.jboss.ejb.plugins.cmp.ejbql.SelectFunction, org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager, boolean[], java.util.List, java.util.List, org.jboss.ejb.plugins.cmp.jdbc.metadata.JDBCQueryMetaData, org.jboss.ejb.GenericEntityObjectFactory, org.jboss.logging.Logger) line: 225
org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery(org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand).execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 144
org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery.execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 155
org.jboss.ejb.plugins.cmp.jdbc.JDBCFindEntityCommand.execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 61
org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 604
org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 315
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 236
org.jboss.ejb.EntityContainer.findSingleObject(javax.transaction.Transaction, java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 1103
org.jboss.ejb.EntityContainer.find(org.jboss.invocation.Invocation) line: 721
sun.reflect.GeneratedMethodAccessor210.invoke(java.lang.Object, java.lang.Object[]) line: not available
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
org.jboss.invocation.Invocation.performCall(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 359
org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 1130
org.jboss.ejb.plugins.cmp.jdbc.JDBCRelationInterceptor(org.jboss.ejb.plugins.AbstractInterceptor).invokeHome(org.jboss.invocation.Invocation) line: 105
org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 203
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 189
org.jboss.ejb.plugins.EntityReentranceInterceptor(org.jboss.ejb.plugins.AbstractInterceptor).invokeHome(org.jboss.invocation.Invocation) line: 105
org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 134
org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 76
org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 43
org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 56
org.jboss.ejb.plugins.TxInterceptorCMT(org.jboss.ejb.plugins.AbstractTxInterceptor).invokeNext(org.jboss.invocation.Invocation, boolean) line: 125
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(org.jboss.invocation.Invocation) line: 350
org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(org.jboss.invocation.Invocation) line: 161
org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 145
org.jboss.ejb.plugins.LogInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 132
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 107
org.jboss.ejb.EntityContainer.internalInvokeHome(org.jboss.invocation.Invocation) line: 514
org.jboss.ejb.EntityContainer(org.jboss.ejb.Container).invoke(org.jboss.invocation.Invocation) line: 975
sun.reflect.GeneratedMethodAccessor105.invoke(java.lang.Object, java.lang.Object[]) line: not available
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(org.jboss.mx.server.Invocation) line: 155
org.jboss.mx.server.Invocation.dispatch() line: 94
org.jboss.mx.server.Invocation.invoke() line: 86
org.jboss.mx.modelmbean.XMBean(org.jboss.mx.server.AbstractMBeanInvoker).invoke(java.lang.String, java.lang.Object[], java.lang.String[]) line: 264
org.jboss.mx.server.MBeanServerImpl.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 659
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 169
org.jboss.invocation.local.LocalInvoker.invoke(org.jboss.invocation.Invocation) line: 118
org.jboss.invocation.InvokerInterceptor.invokeLocal(org.jboss.invocation.Invocation) line: 206
org.jboss.invocation.InvokerInterceptor.invoke(org.jboss.invocation.Invocation) line: 192
org.jboss.proxy.TransactionInterceptor.invoke(org.jboss.invocation.Invocation) line: 61
org.jboss.proxy.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 70
org.jboss.proxy.ejb.HomeInterceptor.invoke(org.jboss.invocation.Invocation) line: 184
org.jboss.proxy.ClientContainer.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 100
$Proxy967.findByPrimaryKey(java.lang.String) line: not available


The handling for this exception never reaches BaseWrapperManagedConnection.checkException() or WrappedConnection.checkException(), let alone any exception sorters.

-----

Here's the second one - a stack trace taken at the time the SQLException is thrown, during a call to an accessor:


Thread [PooledInvokerThread-127.0.0.1-0] (Suspended (exception com.jnetdirect.jsql.JSQLException))
com.jnetdirect.jsql.JSQLException.makeFromDriverError(com.jnetdirect.jsql.JSQLConnection, com.jnetdirect.jsql.IOBuffer, java.lang.String, java.lang.String, boolean) line: not available
com.jnetdirect.jsql.DBComms.transmit(byte, com.jnetdirect.jsql.IOBuffer, int, int, boolean) line: not available
com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.IOBuffer).a(byte, int, boolean) line: not available
com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.JSQLStatement).for(byte) line: not available
com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.JSQLStatement).do(byte) line: not available
com.jnetdirect.jsql.JSQLPreparedStatement.executeQuery() line: not available
org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery() line: 236
org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCCMPFieldBridge, org.jboss.ejb.EntityEnterpriseContext, boolean) line: 177
org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(org.jboss.ejb.EntityEnterpriseContext, boolean) line: 88
org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext, boolean) line: 646
org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 628
org.jboss.ejb.plugins.CMPPersistenceManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 406
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 252
org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(org.jboss.invocation.Invocation) line: 243
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(org.jboss.invocation.Invocation) line: 158
org.jboss.ejb.plugins.EntityReentranceInterceptor.invoke(org.jboss.invocation.Invocation) line: 126
org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(org.jboss.invocation.Invocation) line: 276
org.jboss.ejb.plugins.EntityLockInterceptor.invoke(org.jboss.invocation.Invocation) line: 104
org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(org.jboss.invocation.Invocation) line: 68
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(org.jboss.invocation.Invocation) line: 63
org.jboss.ejb.plugins.TxInterceptorCMT(org.jboss.ejb.plugins.AbstractTxInterceptor).invokeNext(org.jboss.invocation.Invocation, boolean) line: 121
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(org.jboss.invocation.Invocation) line: 378
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(org.jboss.invocation.Invocation) line: 181
org.jboss.ejb.plugins.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 168
org.jboss.ejb.plugins.LogInterceptor.invoke(org.jboss.invocation.Invocation) line: 205
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(org.jboss.invocation.Invocation) line: 136
org.jboss.ejb.EntityContainer.internalInvoke(org.jboss.invocation.Invocation) line: 520
org.jboss.ejb.EntityContainer(org.jboss.ejb.Container).invoke(org.jboss.invocation.Invocation) line: 954
sun.reflect.GeneratedMethodAccessor105.invoke(java.lang.Object, java.lang.Object[]) line: not available
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(org.jboss.mx.server.Invocation) line: 155
org.jboss.mx.server.Invocation.dispatch() line: 94
org.jboss.mx.server.Invocation.invoke() line: 86
org.jboss.mx.modelmbean.XMBean(org.jboss.mx.server.AbstractMBeanInvoker).invoke(java.lang.String, java.lang.Object[], java.lang.String[]) line: 264
org.jboss.mx.server.MBeanServerImpl.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 659
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 169
org.jboss.invocation.local.LocalInvoker.invoke(org.jboss.invocation.Invocation) line: 118
org.jboss.invocation.InvokerInterceptor.invokeLocal(org.jboss.invocation.Invocation) line: 206
org.jboss.invocation.InvokerInterceptor.invoke(org.jboss.invocation.Invocation) line: 192
org.jboss.proxy.TransactionInterceptor.invoke(org.jboss.invocation.Invocation) line: 61
org.jboss.proxy.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 70
org.jboss.proxy.ejb.EntityInterceptor.invoke(org.jboss.invocation.Invocation) line: 112
org.jboss.proxy.ClientContainer.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 100
$Proxy968.getUserId() line: not available


In this case, the container's exception handling does reach WrappedConnection.checkException() - but WrappedConnection.mc is null, so, again, exception sorting is never performed, either by our class or by any of the container's isExceptionFatal() methods.

So far, we have not found any cases with this configuration that invoke or even instantiate the exception sorter when a connection error occurs.

[weston.price]

The first error is an XA exception throws from the XAResource during the 2PC protocol

org.jboss.tm.TransactionImpl$Resource.startResource() line: 2063
 org.jboss.tm.TransactionImpl.enlistResource(javax.transaction.xa.XAResource) line: 581
 org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynch
ronization.enlist() line: 757

XA errors are outside the purview of the sorter as this is not a connection based issue but rather an XA issue. XA exceptions are evaluated in the limited case that certain XA error codes prohibit a connection from being returned to the pool, but that's about it. This is why you are seeing the difference.

[nathanmeyers]

XA errors are outside the purview of the sorter as this is not a connection based issue but rather an XA issue. XA exceptions are evaluated in the limited case that certain XA error codes prohibit a connection from being returned to the pool, but that's about it. This is why you are seeing the difference.

The situation leading to this error was restart of the db server, so I have a pool full of dead connections. I gather from what you're saying that I can't flush the pool from this particular exception-handling path - the best I can hope is that this particular connection isn't returned to the pool. Whether this happens depends on what XA error the driver returns, and I cannot add my own discriminator to this logic if the container doesn't recognize it.

Any insights on the second situation - the one with the null WrappedConnection.mc?

[weston.price]

The situation leading to this error was restart of the db server, so I have a pool full of dead connections.

Your *-ds.xml file does not include a valid connection checker element which means that potentially invalid connections can remain in the pool for use. Take a look at

http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJCAPooling

On how to configure this.

I gather from what you're saying that I can't flush the pool from this particular exception-handling path

I think we are talking about two different things. Connection validation occurs at the pool level prior to a client being given a connection. If your *-ds.xml file includes this, it will be invoked.

The ExceptionSorter path is used to evaluate exceptions that occur on the connection while in use. For XA related errors currently there is no way to intercept this path. There is no reason why this could not be added.

[nathanmeyers]

The situation leading to this error was restart of the db server, so I have a pool full of dead connections.

Your *-ds.xml file does not include a valid connection checker element which means that potentially invalid connections can remain in the pool for use. Take a look at

http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJCAPooling

On how to configure this.

I'm familiar with <check-valid-connection-sql>. The point of this exception-sorter exercise is to get away from the high expense of that approach. Ideally, we want to accept the cost of a single failure - followed by immediate invalidation of the pool - rather than the overhead of very frequent connection checks.

The distinction between XA errors and connection errors is an artifact of how the XA transactions are managed. The bottom line is that a pool full of connections has been rendered bad by a db restart - it's a connection issue any way you cut it, and I'd like to flush the pool the instant that problem is detected.

For reasons you've now explained, I know I can't currently do that in certain parts of the XA lifecycle. And for mysterious reasons (the second scenario), I can't even do that when I should be able to.

[weston.price]

I'm familiar with <check-valid-connection-sql>. The point of this exception-sorter exercise is to get away from the high expense of that approach. Ideally, we want to accept the cost of a single failure - followed by immediate invalidation of the pool - rather than the overhead of very frequent connection checks.

There is no notion of a 'purge policy' in JBoss as this time where the pool is invalidated based on the error state of a *single* failing connection. There is some debate on including this for a future release.

JBoss 4.0.5 includes a background validation mechanism to evaluate pooled connections in the background rather than on retrieval which alleviates the overhead of constant validation.

The distinction between XA errors and connection errors is an artifact of how the XA transactions are managed.

The difference is simply in the origin of the exception. XA based errors are reported from the underlying XA resource which don't currently have a hook to evaluate the error. Again, there is no reason why this couldn't be added.

As for the underlying MC being null, there are a few things that could cause this

1) The connection has been closed and you attempt to reuse the handle

2) The connection has been closed in another thread, and you attempt to reuse the handle

3) You are attempting to reuse the connection across method calls in an EJB and you do not have the spec compliant attribute set to true on the CachedConnectionManager (you would never really want to do this anyway).

4) You are not using connection validation and you have been given an invalid connection from the pool (this would be my guess).

However, I am not seeing this in the stacktrace you posted. If the MC was actually null, you would never be able to execute a prepared statement, we check for this prior to every invocation to prohibit this behavior. I apologize if I am not looking in the right place.