7 Replies Latest reply on Apr 3, 2007 1:46 PM by Weston M. Price

    Are exception-sorter expected to work for xa-datasource?

    Scott Marlow Master

      Are exception-sorter expected to work for xa-datasource?

      The dtd @ http://www.jboss.org/j2ee/dtd/jboss-ds_1_5.dtd seems to think it should work.

      Our debuger breakpoints on exception-filter only seem to get hit on non-xa datasources but not xa-datasources.

      Thanks,
      Scott

        • 1. Re: Are exception-sorter expected to work for xa-datasource?
          Weston M. Price Master

          Exception sorters are transaction independent. Post your *-ds.xml file so we can take a look as well as the version of JBoss and DB that you are using.

          • 2. Re: Are exception-sorter expected to work for xa-datasource?
            nathanmeyers Newbie

            I'm working with Scott on this issue. Here are some more details:

            We're working in JBoss 4.0.4, JDK5, Microsoft SQL Server 2000, Windows XP Pro SP2, JNetDirect 5.532 drivers.

            Here is an excerpt from the datasource declarations:


            <xa-datasource>
            <jndi-name>DefaultDS</jndi-name>
            <track-connection-by-tx>true</track-connection-by-tx>
            <isSameRM-override-value>false</isSameRM-override-value>
            <xa-datasource-class>com.jnetdirect.jsql.JSQLXADataSource</xa-datasource-class>
            <xa-datasource-property name="URL">jdbc:JSQLConnect://nmeyerspc/database=mydb/asciiStringParameters=true</xa-datasource-property>
            <security-domain>DefaultDSRealm</security-domain>
            <no-tx-separate-pools/>
            <max-pool-size>200</max-pool-size>
            <exception-sorter-class-name>util.SQLServerExceptionSorter</exception-sorter-class-name>
            </xa-datasource>


            The problem we're seeing is that the exception sorter is never called or even instantiated. We've seen it work flawlessly with local-tx-datasource, but never with xa-datasource.

            -----

            We're seeing a couple of patterns when connection errors are encountered. Here's the first one - a stack trace taken at the time the SQLException is thrown, during a call to a finder:


            Thread [PooledInvokerThread-127.0.0.1-0] (Suspended (exception com.jnetdirect.jsql.JSQLException))
            com.jnetdirect.jsql.JSQLException.makeFromDriverError(com.jnetdirect.jsql.JSQLConnection, com.jnetdirect.jsql.IOBuffer, java.lang.String, java.lang.String, boolean) line: not available
            com.jnetdirect.jsql.DBComms.transmit(byte, com.jnetdirect.jsql.IOBuffer, int, int, boolean) line: not available
            com.jnetdirect.jsql.JSQLConnection(com.jnetdirect.jsql.IOBuffer).a(byte, int, boolean) line: not available
            com.jnetdirect.jsql.JSQLConnection.new(java.lang.String, java.lang.String, boolean) line: not available
            com.jnetdirect.jsql.JSQLConnection.sendStartTran(java.lang.String) line: not available
            com.jnetdirect.jsql.JSQLXAResource.start(javax.transaction.xa.Xid, int) line: not available
            org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(javax.transaction.xa.Xid, int) line: 117
            org.jboss.tm.TransactionImpl$Resource.startResource() line: 2063
            org.jboss.tm.TransactionImpl.enlistResource(javax.transaction.xa.XAResource) line: 581
            org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist() line: 757
            org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist() line: 548
            org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(org.jboss.resource.connectionmanager.ConnectionListener) line: 323
            org.jboss.resource.connectionmanager.TxConnectionManager(org.jboss.resource.connectionmanager.BaseConnectionManager2).reconnectManagedConnection(org.jboss.resource.connectionmanager.ConnectionListener) line: 501
            org.jboss.resource.connectionmanager.TxConnectionManager(org.jboss.resource.connectionmanager.BaseConnectionManager2).allocateConnection(javax.resource.spi.ManagedConnectionFactory, javax.resource.spi.ConnectionRequestInfo) line: 382
            org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(javax.resource.spi.ManagedConnectionFactory, javax.resource.spi.ConnectionRequestInfo) line: 812
            org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection() line: 88
            org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery(org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand).execute(java.lang.String, java.lang.Object[], int, int, org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCEntityBridge, org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCCMPFieldBridge, org.jboss.ejb.plugins.cmp.ejbql.SelectFunction, org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager, boolean[], java.util.List, java.util.List, org.jboss.ejb.plugins.cmp.jdbc.metadata.JDBCQueryMetaData, org.jboss.ejb.GenericEntityObjectFactory, org.jboss.logging.Logger) line: 225
            org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery(org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand).execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 144
            org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery.execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 155
            org.jboss.ejb.plugins.cmp.jdbc.JDBCFindEntityCommand.execute(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 61
            org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 604
            org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 315
            org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 236
            org.jboss.ejb.EntityContainer.findSingleObject(javax.transaction.Transaction, java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.EntityEnterpriseContext, org.jboss.ejb.GenericEntityObjectFactory) line: 1103
            org.jboss.ejb.EntityContainer.find(org.jboss.invocation.Invocation) line: 721
            sun.reflect.GeneratedMethodAccessor210.invoke(java.lang.Object, java.lang.Object[]) line: not available
            sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
            java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
            org.jboss.invocation.Invocation.performCall(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 359
            org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 1130
            org.jboss.ejb.plugins.cmp.jdbc.JDBCRelationInterceptor(org.jboss.ejb.plugins.AbstractInterceptor).invokeHome(org.jboss.invocation.Invocation) line: 105
            org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 203
            org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 189
            org.jboss.ejb.plugins.EntityReentranceInterceptor(org.jboss.ejb.plugins.AbstractInterceptor).invokeHome(org.jboss.invocation.Invocation) line: 105
            org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 134
            org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 76
            org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 43
            org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 56
            org.jboss.ejb.plugins.TxInterceptorCMT(org.jboss.ejb.plugins.AbstractTxInterceptor).invokeNext(org.jboss.invocation.Invocation, boolean) line: 125
            org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(org.jboss.invocation.Invocation) line: 350
            org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(org.jboss.invocation.Invocation) line: 161
            org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 145
            org.jboss.ejb.plugins.LogInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 132
            org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(org.jboss.invocation.Invocation) line: 107
            org.jboss.ejb.EntityContainer.internalInvokeHome(org.jboss.invocation.Invocation) line: 514
            org.jboss.ejb.EntityContainer(org.jboss.ejb.Container).invoke(org.jboss.invocation.Invocation) line: 975
            sun.reflect.GeneratedMethodAccessor105.invoke(java.lang.Object, java.lang.Object[]) line: not available
            sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
            java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
            org.jboss.mx.interceptor.ReflectedDispatcher.invoke(org.jboss.mx.server.Invocation) line: 155
            org.jboss.mx.server.Invocation.dispatch() line: 94
            org.jboss.mx.server.Invocation.invoke() line: 86
            org.jboss.mx.modelmbean.XMBean(org.jboss.mx.server.AbstractMBeanInvoker).invoke(java.lang.String, java.lang.Object[], java.lang.String[]) line: 264
            org.jboss.mx.server.MBeanServerImpl.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 659
            org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 169
            org.jboss.invocation.local.LocalInvoker.invoke(org.jboss.invocation.Invocation) line: 118
            org.jboss.invocation.InvokerInterceptor.invokeLocal(org.jboss.invocation.Invocation) line: 206
            org.jboss.invocation.InvokerInterceptor.invoke(org.jboss.invocation.Invocation) line: 192
            org.jboss.proxy.TransactionInterceptor.invoke(org.jboss.invocation.Invocation) line: 61
            org.jboss.proxy.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 70
            org.jboss.proxy.ejb.HomeInterceptor.invoke(org.jboss.invocation.Invocation) line: 184
            org.jboss.proxy.ClientContainer.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 100
            $Proxy967.findByPrimaryKey(java.lang.String) line: not available


            The handling for this exception never reaches BaseWrapperManagedConnection.checkException() or WrappedConnection.checkException(), let alone any exception sorters.

            -----

            Here's the second one - a stack trace taken at the time the SQLException is thrown, during a call to an accessor:


            Thread [PooledInvokerThread-127.0.0.1-0] (Suspended (exception com.jnetdirect.jsql.JSQLException))
            com.jnetdirect.jsql.JSQLException.makeFromDriverError(com.jnetdirect.jsql.JSQLConnection, com.jnetdirect.jsql.IOBuffer, java.lang.String, java.lang.String, boolean) line: not available
            com.jnetdirect.jsql.DBComms.transmit(byte, com.jnetdirect.jsql.IOBuffer, int, int, boolean) line: not available
            com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.IOBuffer).a(byte, int, boolean) line: not available
            com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.JSQLStatement).for(byte) line: not available
            com.jnetdirect.jsql.JSQLPreparedStatement(com.jnetdirect.jsql.JSQLStatement).do(byte) line: not available
            com.jnetdirect.jsql.JSQLPreparedStatement.executeQuery() line: not available
            org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery() line: 236
            org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(org.jboss.ejb.plugins.cmp.jdbc.bridge.JDBCCMPFieldBridge, org.jboss.ejb.EntityEnterpriseContext, boolean) line: 177
            org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(org.jboss.ejb.EntityEnterpriseContext, boolean) line: 88
            org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext, boolean) line: 646
            org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 628
            org.jboss.ejb.plugins.CMPPersistenceManager.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 406
            org.jboss.resource.connectionmanager.CachedConnectionInterceptor.loadEntity(org.jboss.ejb.EntityEnterpriseContext) line: 252
            org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(org.jboss.invocation.Invocation) line: 243
            org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(org.jboss.invocation.Invocation) line: 158
            org.jboss.ejb.plugins.EntityReentranceInterceptor.invoke(org.jboss.invocation.Invocation) line: 126
            org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(org.jboss.invocation.Invocation) line: 276
            org.jboss.ejb.plugins.EntityLockInterceptor.invoke(org.jboss.invocation.Invocation) line: 104
            org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(org.jboss.invocation.Invocation) line: 68
            org.jboss.ejb.plugins.CallValidationInterceptor.invoke(org.jboss.invocation.Invocation) line: 63
            org.jboss.ejb.plugins.TxInterceptorCMT(org.jboss.ejb.plugins.AbstractTxInterceptor).invokeNext(org.jboss.invocation.Invocation, boolean) line: 121
            org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(org.jboss.invocation.Invocation) line: 378
            org.jboss.ejb.plugins.TxInterceptorCMT.invoke(org.jboss.invocation.Invocation) line: 181
            org.jboss.ejb.plugins.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 168
            org.jboss.ejb.plugins.LogInterceptor.invoke(org.jboss.invocation.Invocation) line: 205
            org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(org.jboss.invocation.Invocation) line: 136
            org.jboss.ejb.EntityContainer.internalInvoke(org.jboss.invocation.Invocation) line: 520
            org.jboss.ejb.EntityContainer(org.jboss.ejb.Container).invoke(org.jboss.invocation.Invocation) line: 954
            sun.reflect.GeneratedMethodAccessor105.invoke(java.lang.Object, java.lang.Object[]) line: not available
            sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
            java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
            org.jboss.mx.interceptor.ReflectedDispatcher.invoke(org.jboss.mx.server.Invocation) line: 155
            org.jboss.mx.server.Invocation.dispatch() line: 94
            org.jboss.mx.server.Invocation.invoke() line: 86
            org.jboss.mx.modelmbean.XMBean(org.jboss.mx.server.AbstractMBeanInvoker).invoke(java.lang.String, java.lang.Object[], java.lang.String[]) line: 264
            org.jboss.mx.server.MBeanServerImpl.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 659
            org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(javax.management.ObjectName, java.lang.String, java.lang.Object[], java.lang.String[]) line: 169
            org.jboss.invocation.local.LocalInvoker.invoke(org.jboss.invocation.Invocation) line: 118
            org.jboss.invocation.InvokerInterceptor.invokeLocal(org.jboss.invocation.Invocation) line: 206
            org.jboss.invocation.InvokerInterceptor.invoke(org.jboss.invocation.Invocation) line: 192
            org.jboss.proxy.TransactionInterceptor.invoke(org.jboss.invocation.Invocation) line: 61
            org.jboss.proxy.SecurityInterceptor.invoke(org.jboss.invocation.Invocation) line: 70
            org.jboss.proxy.ejb.EntityInterceptor.invoke(org.jboss.invocation.Invocation) line: 112
            org.jboss.proxy.ClientContainer.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 100
            $Proxy968.getUserId() line: not available


            In this case, the container's exception handling does reach WrappedConnection.checkException() - but WrappedConnection.mc is null, so, again, exception sorting is never performed, either by our class or by any of the container's isExceptionFatal() methods.

            So far, we have not found any cases with this configuration that invoke or even instantiate the exception sorter when a connection error occurs.

            • 3. Re: Are exception-sorter expected to work for xa-datasource?
              Weston M. Price Master

              The first error is an XA exception throws from the XAResource during the 2PC protocol

              org.jboss.tm.TransactionImpl$Resource.startResource() line: 2063
               org.jboss.tm.TransactionImpl.enlistResource(javax.transaction.xa.XAResource) line: 581
               org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynch
              ronization.enlist() line: 757
              


              XA errors are outside the purview of the sorter as this is not a connection based issue but rather an XA issue. XA exceptions are evaluated in the limited case that certain XA error codes prohibit a connection from being returned to the pool, but that's about it. This is why you are seeing the difference.



              • 4. Re: Are exception-sorter expected to work for xa-datasource?
                nathanmeyers Newbie

                 

                "weston.price@jboss.com" wrote:

                XA errors are outside the purview of the sorter as this is not a connection based issue but rather an XA issue. XA exceptions are evaluated in the limited case that certain XA error codes prohibit a connection from being returned to the pool, but that's about it. This is why you are seeing the difference.


                The situation leading to this error was restart of the db server, so I have a pool full of dead connections. I gather from what you're saying that I can't flush the pool from this particular exception-handling path - the best I can hope is that this particular connection isn't returned to the pool. Whether this happens depends on what XA error the driver returns, and I cannot add my own discriminator to this logic if the container doesn't recognize it.

                Any insights on the second situation - the one with the null WrappedConnection.mc?

                • 5. Re: Are exception-sorter expected to work for xa-datasource?
                  Weston M. Price Master

                   


                  The situation leading to this error was restart of the db server, so I have a pool full of dead connections.


                  Your *-ds.xml file does not include a valid connection checker element which means that potentially invalid connections can remain in the pool for use. Take a look at

                  http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJCAPooling

                  On how to configure this.


                  I gather from what you're saying that I can't flush the pool from this particular exception-handling path


                  I think we are talking about two different things. Connection validation occurs at the pool level prior to a client being given a connection. If your *-ds.xml file includes this, it will be invoked.

                  The ExceptionSorter path is used to evaluate exceptions that occur on the connection while in use. For XA related errors currently there is no way to intercept this path. There is no reason why this could not be added.



                  • 6. Re: Are exception-sorter expected to work for xa-datasource?
                    nathanmeyers Newbie

                     

                    "weston.price@jboss.com" wrote:

                    The situation leading to this error was restart of the db server, so I have a pool full of dead connections.


                    Your *-ds.xml file does not include a valid connection checker element which means that potentially invalid connections can remain in the pool for use. Take a look at

                    http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJCAPooling

                    On how to configure this.


                    I'm familiar with <check-valid-connection-sql>. The point of this exception-sorter exercise is to get away from the high expense of that approach. Ideally, we want to accept the cost of a single failure - followed by immediate invalidation of the pool - rather than the overhead of very frequent connection checks.

                    The distinction between XA errors and connection errors is an artifact of how the XA transactions are managed. The bottom line is that a pool full of connections has been rendered bad by a db restart - it's a connection issue any way you cut it, and I'd like to flush the pool the instant that problem is detected.

                    For reasons you've now explained, I know I can't currently do that in certain parts of the XA lifecycle. And for mysterious reasons (the second scenario), I can't even do that when I should be able to.

                    • 7. Re: Are exception-sorter expected to work for xa-datasource?
                      Weston M. Price Master

                       


                      I'm familiar with <check-valid-connection-sql>. The point of this exception-sorter exercise is to get away from the high expense of that approach. Ideally, we want to accept the cost of a single failure - followed by immediate invalidation of the pool - rather than the overhead of very frequent connection checks.


                      There is no notion of a 'purge policy' in JBoss as this time where the pool is invalidated based on the error state of a *single* failing connection. There is some debate on including this for a future release.

                      JBoss 4.0.5 includes a background validation mechanism to evaluate pooled connections in the background rather than on retrieval which alleviates the overhead of constant validation.


                      The distinction between XA errors and connection errors is an artifact of how the XA transactions are managed.


                      The difference is simply in the origin of the exception. XA based errors are reported from the underlying XA resource which don't currently have a hook to evaluate the error. Again, there is no reason why this couldn't be added.

                      As for the underlying MC being null, there are a few things that could cause this

                      1) The connection has been closed and you attempt to reuse the handle

                      2) The connection has been closed in another thread, and you attempt to reuse the handle

                      3) You are attempting to reuse the connection across method calls in an EJB and you do not have the spec compliant attribute set to true on the CachedConnectionManager (you would never really want to do this anyway).

                      4) You are not using connection validation and you have been given an invalid connection from the pool (this would be my guess).

                      However, I am not seeing this in the stacktrace you posted. If the MC was actually null, you would never be able to execute a prepared statement, we check for this prior to every invocation to prohibit this behavior. I apologize if I am not looking in the right place.