trsanction aborted/rollbacked with un-understand exception

[bondchan921]

I'm using Jboss5.1.0 and oracle 11.2.0.3.0.

1)does the first exception cause the transaction aborted? then causing the TransactionRollBackException

2)what's the 1st exception means?

/***************/

2014-02-21 11:20:46,711 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (WorkManager(2)-19:) registering connection from org.jboss.resource.connectionmanager.TxConnectionManager@9437a04, connection : org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@314b2c15, key: com.lombardrisk.collateral.persistence.statement.ColStatementCMP$Proxy@530d6e0b

2014-02-21 11:20:46,713 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (WorkManager(2)-19:) unregistering connection from org.jboss.resource.connectionmanager.TxConnectionManager@9437a04, object: org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@314b2c15, key: com.lombardrisk.collateral.persistence.statement.ColStatementCMP$Proxy@530d6e0b

2014-02-21 11:20:46,716 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) putting ManagedConnection back into pool kill=true cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@5d14ea11[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@81331d5 handles=0 lastUse=1392999007317 permit=true trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@1f180dec context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@643a10e6 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@242c9b17 txSync=TxSync954745361{tx=TransactionImple < ac, BasicAction: a201e41:e4dc:530778a7:24fa status: ActionStatus.PREPARING > wasTrackByTx=true enlisted=true}]

2014-02-21 11:20:46,716 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Destroying returned connection org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@5d14ea11[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@81331d5 handles=0 lastUse=1392999007317 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@1f180dec context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@643a10e6 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@242c9b17 txSync=TxSync954745361{tx=TransactionImple < ac, BasicAction: a201e41:e4dc:530778a7:24fa status: ActionStatus.PREPARING > wasTrackByTx=true enlisted=true}]

2014-02-21 11:20:46,716 TRACE [org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory] (FirmamentScheduler_Worker-0:) Ignored error during rollback:

java.sql.SQLRecoverableException: Closed Connection

  at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:3694)

  at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.destroy(BaseWrapperManagedConnection.java:303)

  at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.doDestroy(InternalManagedConnectionPool.java:664)

  at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.returnConnection(InternalManagedConnectionPool.java:395)

  at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.returnConnection(JBossManagedConnectionPool.java:724)

  at org.jboss.resource.connectionmanager.BaseConnectionManager2.returnManagedConnection(BaseConnectionManager2.java:463)

  at org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource.commit(TxConnectionManager.java:1062)

  at com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.commit(XAOnePhaseResource.java:110)

  at com.arjuna.ats.arjuna.LastResourceRecord.topLevelPrepare(LastResourceRecord.java:170)

  at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2919)

  at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2876)

  at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2419)

  at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1790)

  at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94)

  at com.arjuna.ats.arjuna.AtomicAction.end(AtomicAction.java:216)

  at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commit(TransactionImple.java:242)

  at org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java:501)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:361)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

  at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

  at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

  at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

  at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)

  at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)

  at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)

  at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)

  at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

  at org.jboss.proxy.ejb.SecurityContextInterceptor.invoke(SecurityContextInterceptor.java:64)

  at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)

  at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)

  at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

  at $Proxy3358.importData(Unknown Source)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.importData(ColImportHugeTradesTask.java:53)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.importHugeFile(ColImportHugeTradesTaskBase.java:353)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.executeImport(ColImportHugeTradesTaskBase.java:170)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.<init>(ColImportHugeTradesTask.java:33)

  at com.lombardrisk.collateral.scheduler.ColImportTradesJob.importHugeTask(ColImportTradesJob.java:66)

  at com.lombardrisk.collateral.scheduler.ColGenericImportTradesJob.executeJob(ColGenericImportTradesJob.java:218)

  at com.lombardrisk.f3.job.RunAsJobSupport.execute(RunAsJobSupport.java:179)

  at org.quartz.core.JobRunShell.run(JobRunShell.java:203)

  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

2014-02-21 11:20:46,718 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Returning connection to pool org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@5d14ea11[state=DESTROYED mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@81331d5 handles=0 lastUse=1392999007317 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@1f180dec context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@643a10e6 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@242c9b17 txSync=TxSync954745361{tx=TransactionImple < ac, BasicAction: a201e41:e4dc:530778a7:24fa status: ActionStatus.PREPARING > wasTrackByTx=true enlisted=true}] [InUse/Available/Max]: [1/9/10]

2014-02-21 11:20:46,735 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (FirmamentScheduler_Worker-0:) [com.arjuna.ats.arjuna.coordinator.BasicAction_36] - BasicAction.End() - prepare phase of action-id a201e41:e4dc:530778a7:24fa failed.

2014-02-21 11:20:46,735 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (FirmamentScheduler_Worker-0:) [com.arjuna.ats.arjuna.coordinator.BasicAction_38] - Action Aborting

2014-02-21 11:20:51,486 WARN  [com.arjuna.ats.jta.logging.logger] (FirmamentScheduler_Worker-0:) XAOnePhaseResource.rollback(< 131075, 28, 26, 49459750484910152495810152100995853514855555697555850521029797504849101524958101521009958535148555556975558505210299 >) wrong xid in rollback: expected: null, got: < 131075, 28, 26, 49459750484910152495810152100995853514855555697555850521029797504849101524958101521009958535148555556975558505210299 >

2014-02-21 11:20:51,497 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (FirmamentScheduler_Worker-0:) [com.arjuna.ats.arjuna.coordinator.BasicAction_54] - Top-level abort of action a201e41:e4dc:530778a7:24fa received TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.LASTRESOURCE>

2014-02-21 11:20:51,693 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) putting ManagedConnection back into pool kill=false cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@51de4e31[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 handles=0 lastUse=1392999004065 permit=true trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@17336201 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@2e673b8d xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 txSync=null]

2014-02-21 11:20:51,693 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Returning connection to pool org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@51de4e31[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 handles=0 lastUse=1393003251693 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@17336201 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@2e673b8d xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 txSync=null] [InUse/Available/Max]: [2/18/20]

2014-02-21 11:20:51,698 ERROR [org.jboss.ejb.plugins.LogInterceptor] (FirmamentScheduler_Worker-0:) TransactionRolledbackException in method: public abstract com.lombardrisk.common.etl.ImportStatus com.lombardrisk.collateral.ColTradeFeedServiceRemote.importData(java.util.Collection,java.lang.String,java.io.File,java.lang.String,java.lang.Integer,com.lombardrisk.f3.locking.StatementVersionStamp) throws java.lang.Exception,java.rmi.RemoteException, causedBy:

javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] Could not commit transaction.

  at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commit(TransactionImple.java:257)

  at org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java:501)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:361)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

  at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

  at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

  at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

  at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)

  at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)

  at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)

  at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)

  at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

  at org.jboss.proxy.ejb.SecurityContextInterceptor.invoke(SecurityContextInterceptor.java:64)

  at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)

  at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)

  at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

  at $Proxy3358.importData(Unknown Source)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.importData(ColImportHugeTradesTask.java:53)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.importHugeFile(ColImportHugeTradesTaskBase.java:353)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.executeImport(ColImportHugeTradesTaskBase.java:170)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.<init>(ColImportHugeTradesTask.java:33)

  at com.lombardrisk.collateral.scheduler.ColImportTradesJob.importHugeTask(ColImportTradesJob.java:66)

  at com.lombardrisk.collateral.scheduler.ColGenericImportTradesJob.executeJob(ColGenericImportTradesJob.java:218)

  at com.lombardrisk.f3.job.RunAsJobSupport.execute(RunAsJobSupport.java:179)

  at org.quartz.core.JobRunShell.run(JobRunShell.java:203)

  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

2014-02-21 11:20:51,699 ERROR [com.lombardrisk.collateral.etl.ColImportHugeTradesTask] (FirmamentScheduler_Worker-0:) org.jboss.tm.JBossTransactionRolledbackException: null; nested exception is:

  javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] Could not commit transaction.; - nested throwable: (javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] Could not commit transaction.)

2014-02-21 11:20:51,699 DEBUG [com.lombardrisk.collateral.etl.ColImportHugeTradesTask] (FirmamentScheduler_Worker-0:) import data cost:4243539

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) old stack for key: com.lombardrisk.f3.messages.MessageServiceBean@2081cc10

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) popped object: org.jboss.resource.connectionmanager.CachedConnectionManager$KeyConnectionAssociation@317d392a

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) old stack for key: com.lombardrisk.f3.messages.MessageServiceBean@2081cc10

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) old stack for key: com.lombardrisk.f3.messages.ActiveMessageCMP$Proxy@616be1f9

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) supplying ManagedConnection from pool: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@51de4e31[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 handles=0 lastUse=1393003251693 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@17336201 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@2e673b8d xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 txSync=null]

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Got connection from pool tracked by transaction org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@51de4e31[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 handles=0 lastUse=1393003251693 permit=true trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@17336201 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@2e673b8d xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 txSync=null] tx=TransactionImple < ac, BasicAction: a201e41:e4dc:530778a7:1f8d6 status: ActionStatus.RUNNING > [InUse/Available/Max]: [3/17/20]

2014-02-21 11:20:51,700 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Using connection from pool tracked by transaction org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@51de4e31[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 handles=0 lastUse=1393003251693 permit=true trackByTx=true mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@17336201 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@2e673b8d xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@59df9cf4 txSync=null] tx=TransactionImple < ac, BasicAction: a201e41:e4dc:530778a7:1f8d6 status: ActionStatus.RUNNING > [InUse/Available/Max]: [3/17/20]

2014-02-21 11:20:51,702 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) registering connection from org.jboss.resource.connectionmanager.TxConnectionManager@30a59324, connection : org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@71c0d466, key: com.lombardrisk.f3.messages.ActiveMessageCMP$Proxy@616be1f9

2014-02-21 11:20:51,704 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) unregistering connection from org.jboss.resource.connectionmanager.TxConnectionManager@30a59324, object: org.jboss.resource.adapter.jdbc.jdk6.WrappedConnectionJDK6@71c0d466, key: com.lombardrisk.f3.messages.ActiveMessageCMP$Proxy@616be1f9

[bondchan921]

Anyone had this issue before ?

[paul.robinson]

Bond,

Are you able to upgrade to WildFly 8 or JBoss EAP 6.2.0 (or at least try your application on it)? We have some new tooling that should help you understand why the transaction aborted:

http://jbossts.blogspot.co.uk/2014/01/narayana-transaction-analyser-100alpha1.html

Paul.

[tomjenkinson]

Hi,

+1 to trying this on something EAP 6.2 or WildFly.

It looks like the commit of the 1PC failed inside Oracle at some point to which JCA then looks to have tried to rollback the connection, traced that error and returned an error to JBTM. The stack trace itself looks like it happened _after_ aTxConnectionManager$LocalXAResource.commit(TxConnectionManager.java:1062) but you would need to ask jesper.pedersen about that.

Tom

[bondchan921]

Thanks Paul, our application have dependency with jboss-cache 1.4.x, upgrade to EAP 6 would be a large job.

[bondchan921]

Thanks Tom,

from How to fall back to the legacy JBossTM fast in-memory transa

the action abort was caused by the last resource which only supports commit(not 2-pc) failed at prepare phase, which confused me, we have used resources: jboss message, jboss-cache(1.4.1.sp12), database resource.

jboss-cache is not 2pc Chapter 2. JBoss Cache - Core , is this last resource? causing action aborted ? then cause oracle rollback connection ? then cause the TransactionRollbackException

and Is there any useful information we can found at oracle server ?

[tomjenkinson]

Hi Bond,

The Oracle (database) resource is being used in 1PC mode but wrapped in a 2PC facade. You need to work out why the commit didn't work. You need a breakpoint in JCA: org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource.commit track that through and see what the matter is.

The best oracle resource is going to be their website.

Tom

[bondchan921]

Hi Tom,

1)But we config oracle datasource as xa-datasource, do you mean arjuna still treat it as last-resource as well?

2)From another run have another error, the old exceptions still exists 2 seconds after this one,  one of the last-resouce may be Jboss-message(jboss 5.1.0)'s local-xa-datasource, do you think another one is jboss cache or Oracle xa-datasource

3)For Jboss-message to be 2pc, we need change local-xa-datasource to xa-datasource ?(Multiple1PC)

/********************/

2014-02-23 19:59:30,336 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (FirmamentScheduler_Worker-0:) [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.disallow] [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.disallow] Adding multiple last resources is disallowed. Current resource is org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@67a17bd5

2014-02-23 19:59:30,342 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) putting ManagedConnection back into pool kill=false cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@6a273588[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@342baa2a handles=0 lastUse=1393207169535 permit=true trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@7cf20761 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@1eb0c628 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@67a17bd5 txSync=null]

2014-02-23 19:59:30,342 TRACE [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] (FirmamentScheduler_Worker-0:) Returning connection to pool org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@6a273588[state=NORMAL mc=org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@342baa2a handles=0 lastUse=1393207170342 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubject@7cf20761 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@1eb0c628 xaResource=org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@67a17bd5 txSync=null] [InUse/Available/Max]: [0/10/10]

2014-02-23 19:59:30,346 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-0:) org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >))

2014-02-23 19:59:30,347 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) old stack for key: com.lombardrisk.collateral.persistence.statement.StatementPersistenceManagerBean@7fd086b4

2014-02-23 19:59:30,347 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) popped object: org.jboss.resource.connectionmanager.CachedConnectionManager$KeyConnectionAssociation@289806b1

2014-02-23 19:59:30,348 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-0:) org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >)

2014-02-23 19:59:30,348 TRACE [org.jboss.resource.connectionmanager.CachedConnectionManager] (FirmamentScheduler_Worker-0:) old stack for key: com.lombardrisk.collateral.persistence.statement.StatementPersistenceManagerBean@7fd086b4

2014-02-23 19:59:30,348 ERROR [com.lombardrisk.collateral.persistence.statement.StatementPersistenceManagerBean] (FirmamentScheduler_Worker-0:) Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >)

org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >)

  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:95)

  at com.lombardrisk.common.jdbc.SimpleJDBCTemplate.executeBatchPreparedStatement(SimpleJDBCTemplate.java:240)

  at com.lombardrisk.collateral.persistence.statement.StatementPersistenceManagerBean.createStatementCalcReq(StatementPersistenceManagerBean.java:539)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.invocation.Invocation.performCall(Invocation.java:386)

  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)

  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)

  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)

  at com.lombardrisk.f3.audit.AuditTrailInterceptor.invoke(AuditTrailInterceptor.java:233)

  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:436)

  at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

  at $Proxy3351.createStatementCalcReq(Unknown Source)

  at com.lombardrisk.collateral.statement.StatementServiceBean.notifyStatamentsFeedChangeInternal(StatementServiceBean.java:1387)

  at com.lombardrisk.collateral.statement.StatementServiceBean.notifyStatamentsFeedChange(StatementServiceBean.java:1338)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.invocation.Invocation.performCall(Invocation.java:386)

  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)

  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)

  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)

  at com.lombardrisk.f3.audit.AuditTrailInterceptor.invoke(AuditTrailInterceptor.java:233)

  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:404)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:436)

  at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

  at $Proxy3561.notifyStatamentsFeedChange(Unknown Source)

  at com.lombardrisk.collateral.calculationengine.util.FeedMessageSender.triggerStatusChange(FeedMessageSender.java:163)

  at com.lombardrisk.collateral.util.TradeFeedUtils.triggerCalc(TradeFeedUtils.java:207)

  at com.lombardrisk.collateral.ColTradeServiceBean.doImport(ColTradeServiceBean.java:467)

  at com.lombardrisk.collateral.ColTradeServiceBean.importData(ColTradeServiceBean.java:662)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.invocation.Invocation.performCall(Invocation.java:386)

  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)

  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)

  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)

  at com.lombardrisk.f3.audit.AuditTrailInterceptor.invoke(AuditTrailInterceptor.java:233)

  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:436)

  at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)

  at $Proxy3456.importData(Unknown Source)

  at com.lombardrisk.collateral.ColTradeFeedServiceBean.doImport(ColTradeFeedServiceBean.java:477)

  at com.lombardrisk.collateral.ColTradeFeedServiceBean.importData(ColTradeFeedServiceBean.java:450)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.invocation.Invocation.performCall(Invocation.java:386)

  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)

  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)

  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)

  at com.lombardrisk.f3.audit.AuditTrailInterceptor.invoke(AuditTrailInterceptor.java:233)

  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)

  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

  at com.lombardrisk.f3.accesscontrol.JBossEJBInterceptor.invoke(JBossEJBInterceptor.java:140)

  at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:228)

  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)

  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)

  at org.jboss.ejb.Container.invoke(Container.java:1029)

  at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)

  at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

  at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

  at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

  at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

  at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)

  at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)

  at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)

  at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)

  at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

  at org.jboss.proxy.ejb.SecurityContextInterceptor.invoke(SecurityContextInterceptor.java:64)

  at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)

  at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)

  at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

  at $Proxy3358.importData(Unknown Source)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.importData(ColImportHugeTradesTask.java:53)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.importHugeFile(ColImportHugeTradesTaskBase.java:353)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTaskBase.executeImport(ColImportHugeTradesTaskBase.java:170)

  at com.lombardrisk.collateral.etl.ColImportHugeTradesTask.<init>(ColImportHugeTradesTask.java:33)

  at com.lombardrisk.collateral.scheduler.ColImportTradesJob.importHugeTask(ColImportTradesJob.java:66)

  at com.lombardrisk.collateral.scheduler.ColGenericImportTradesJob.executeJob(ColGenericImportTradesJob.java:218)

  at com.lombardrisk.f3.job.RunAsJobSupport.execute(RunAsJobSupport.java:179)

  at org.quartz.core.JobRunShell.run(JobRunShell.java:203)

  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

Caused by: javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a201e41:80ac:530a95dd:18a55 status: ActionStatus.ABORT_ONLY >

  at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:370)

  at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

  at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

  ... 126 more

[tomjenkinson]

Hi Bond:

This tells me that the oracle datasource is not being being used in XA mode:

{code}

  at com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.commit(XAOnePhaseResource.java:110)

  at com.arjuna.ats.arjuna.LastResourceRecord.topLevelPrepare(LastResourceRecord.java:170)

  at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2919)
{code}

I am not familiar with JBoss 5 but post your config and someone on the forum might be able to help you.

Also, the error message tells me that you have configured multiple 1PC resources in a 2PC transaction. This is dangerous and not recommended.

Personally, I would try to configure all the resources to be proper XA ones, but most importantly I would try this on WildFly.

Tom

[bondchan921]

Thanks Tom,

See my datasource config file, so the JbossmqDS datasource is 1PC resource?

<datasources>

  <xa-datasource>

    <jndi-name>DefaultDS</jndi-name>

    <track-connection-by-tx>true</track-connection-by-tx>

    <isSameRM-override-value>false</isSameRM-override-value>

  <xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>

  <security-domain>DefaultDSRealm</security-domain>

    <xa-datasource-property name="URL">jdbc:oracle:thin:@xxxxxxxx:1521:dev2</xa-datasource-property>

    <xa-datasource-property name="ImplicitCachingEnabled">true</xa-datasource-property>

    <xa-datasource-property name="MaxStatements">50</xa-datasource-property>

  <min-pool-size>5</min-pool-size>

  <max-pool-size>20</max-pool-size>

  <connection-property name="oracle.jdbc.Trace">true</connection-property> 

    <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>

    <metadata>

      <type-mapping>Oracle9i</type-mapping>

    </metadata>

  <new-connection-sql>select callAlterSession('nnnnnn','${jboss.db.caseSensitive:true}') from dual</new-connection-sql>

  </xa-datasource>

  <!-- -->

  <local-tx-datasource>

    <jndi-name>JbossmqDS</jndi-name>

    <connection-url>jdbc:oracle:thin:@xxxxxxx:1521:dev2</connection-url>

    <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>   

    <security-domain>JbossmqDSRealm</security-domain>

    <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

  <new-connection-sql>select callAlterSession('nnnnnn','${jboss.db.caseSensitive:true}') from dual</new-connection-sql>

  <connection-property name="oracle.jdbc.Trace">true</connection-property> 

  </local-tx-datasource>

   <!---->

  <local-tx-datasource>

    <jndi-name>LoginDS</jndi-name>

    <connection-url>jdbc:oracle:thin:@xxxxxxxx:1521:dev2</connection-url>

    <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>

    <security-domain>LoginDSRealm</security-domain>

    <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>

  <new-connection-sql>select callAlterSession('nnnnnn','${jboss.db.caseSensitive:true}') from dual</new-connection-sql>

  <connection-property name="oracle.jdbc.Trace">true</connection-property> 

  </local-tx-datasource>

[tomjenkinson]

Certainly looks that way. I would try to configure JbossmqDS as an XA datasource and see how you get on.

[bondchan921]

Thanks Tom,  I will let Client try on their UAT.

By enabling JDBC Logging, we found the following exception thrown several mins before the TransactionRollback, Do you think it's related?

2014-02-25 15:28:38,284 SEVERE [oracle.jdbc.driver] (FirmamentScheduler_Worker-0:) Throwing SQLException: java.net.SocketException: Connection reset

2014-02-25 15:28:38,286 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-0:) java.sql.SQLRecoverableException: IO Error: Connection reset

[tomjenkinson]

Certainly looks to be. If the connection dies during a transaction it won't be able to commit

[bondchan921]

Hi Tom,

We have get the test result from client, with re-config the JbossmqDS and LoginDS to XA resource, get different exception now.

2014-02-27 10:57:28,498 CONFIG [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) SQL: select MethodName from AuditTrailMethodLog where ModuleName=? and BeanName=?

2014-02-27 10:57:28,499 SEVERE [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) Throwing SQLException: java.net.SocketException: Connection reset

2014-02-27 10:57:28,502 SEVERE [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) Throwing SQLException: 115

2014-02-27 10:57:28,503 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-4:) java.sql.SQLRecoverableException: IO Error: Connection reset

2014-02-27 10:57:28,505 SEVERE [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) Throwing SQLException: 8

2014-02-27 10:57:28,506 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-4:) java.sql.SQLRecoverableException: Closed Connection

2014-02-27 10:57:28,508 DEBUG [org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreEntityCommand.ColFeedStatus] (FirmamentScheduler_Worker-4:) Executing SQL: UPDATE COLFEEDSTATUS SET loadStatus=?, totalLoaded=?, totalFailed=?, unknownCpty=?, disabledAgreement=?, notSelected=?, invalidDataW=?, invalidDataE=?, refeed=?, genLock=? WHERE id=? AND genLock=?

2014-02-27 10:57:28,509 CONFIG [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) SQL: UPDATE COLFEEDSTATUS SET loadStatus=?, totalLoaded=?, totalFailed=?, unknownCpty=?, disabledAgreement=?, notSelected=?, invalidDataW=?, invalidDataE=?, refeed=?, genLock=? WHERE id=? AND genLock=?

2014-02-27 10:57:28,512 DEBUG [org.jboss.ejb.plugins.cmp.jdbc.JDBCJBossQLQuery.User#findAll] (FirmamentScheduler_Worker-4:) Executing SQL: SELECT t0_u.id, t0_u.PrincipalId, t0_u.passwd, t0_u.enabled, t0_u.failedAttempts, t0_u.maxFailures, t0_u.pwdBirth, t0_u.maxPwdAge, t0_u.allowPwdReuse, t0_u.mustChangePwd, t0_u.genLock FROM PRINCIPALS t0_u ORDER BY t0_u.PrincipalId ASC

2014-02-27 10:57:28,513 CONFIG [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) SQL: SELECT t0_u.id, t0_u.PrincipalId, t0_u.passwd, t0_u.enabled, t0_u.failedAttempts, t0_u.maxFailures, t0_u.pwdBirth, t0_u.maxPwdAge, t0_u.allowPwdReuse, t0_u.mustChangePwd, t0_u.genLock FROM PRINCIPALS t0_u ORDER BY t0_u.PrincipalId ASC

2014-02-27 10:57:28,634 SEVERE [oracle.jdbc.driver] (FirmamentScheduler_Worker-4:) Throwing SQLException: 8

2014-02-27 10:57:28,635 SEVERE [oracle.jdbc.xa] (FirmamentScheduler_Worker-4:) Throwing OracleXAException: 17008

2014-02-27 10:57:28,640 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (FirmamentScheduler_Worker-4:) [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] [com.arjuna.ats.internal.jta.transaction.arjunacore.delistresource] TransactionImple.delistResource - caught exception during delist : XAException.XAER_RMFAIL

2014-02-27 10:57:28,643 DEBUG [org.jboss.util.NestedThrowable] (FirmamentScheduler_Worker-4:) org.jboss.util.NestedThrowable.parentTraceEnabled=true

2014-02-27 10:57:28,643 DEBUG [org.jboss.util.NestedThrowable] (FirmamentScheduler_Worker-4:) org.jboss.util.NestedThrowable.nestedTraceEnabled=false

2014-02-27 10:57:28,643 DEBUG [org.jboss.util.NestedThrowable] (FirmamentScheduler_Worker-4:) org.jboss.util.NestedThrowable.detectDuplicateNesting=true

2014-02-27 10:57:28,643 ERROR [com.lombardrisk.f3.audit.AuditTrailConfiguration] (FirmamentScheduler_Worker-4:) org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Can't enlist - already a tx!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Can't enlist - already a tx!))

2014-02-27 10:57:28,648 ERROR [org.jboss.ejb.plugins.LogInterceptor] (FirmamentScheduler_Worker-4:) TransactionRolledbackLocalException in method: public abstract boolean com.lombardrisk.f3.useradmin.UserLocal.getEnabled(), causedBy:

javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.syncwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.syncwhenaborted] Can't register synchronization because the transaction is in aborted state

  at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.registerSynchronizationImple(TransactionImple.java:476)

[tomjenkinson]

It looks like the underlying issue affects both XA and none-XA. Its the same issue: "Connection reset"

You need some more debugging to find out why the connection is being reset. It could be a firewall or something like that, or the other side could be closing the connection due to failure.