9 Replies Latest reply on Aug 2, 2018 6:14 PM by Ali Alwadi

    Transaction Reaper Worker is causing a deadlock

    Ali Alwadi Newbie

      Hi All,

       

      We have a client Production system which holds the queues to do the back office batch processing. We have a dead-lock situation when apparently one queue threads is locked waiting for Transaction Reaper thread to commit/Rollback the transaction in order to close it off.

      The issue is similar to https://developer.jboss.org/thread/219642, except in our case I couldn't find the relevant code in our application.

      Please see below the relevant part of the thread dump. Unfortunately, as this is a production environment, I couldn't get a heapdump,

      Found one Java-level deadlock:
      =============================
      "GENQ-2":
        waiting to lock monitor 0x00007f8d6620c718 (object 0x00000006c33457f0, a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction),
        which is held by "Transaction Reaper Worker 0"
      "Transaction Reaper Worker 0":
        waiting for ownable synchronizer 0x0000000739b40b88, (a java.util.concurrent.locks.ReentrantLock$FairSync),
        which is held by "GENQ-2"

      Java stack information for the threads listed above:
      ===================================================
      "GENQ-2":
        at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1573)
        - waiting to lock <0x00000006c33457f0> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118)
        at com.arjuna.ats.arjuna.AtomicAction.abort(AtomicAction.java:188)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.rollbackAndDisassociate(TransactionImple.java:1247)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.rollback(BaseTransaction.java:134)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.rollback(BaseTransactionManagerDelegate.java:114)
        at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.rollback(ServerVMClientUserTransaction.java:175)
        at org.sqlb.sql.transaction.JEETransaction.rollback(JEETransaction.java:71)
        at org.sqlb.call.delegate.server.CallDelegateBase.run(CallDelegateBase.java:317)
        at org.sqlb.call.delegate.server.CallDelegateBase.runVoid(CallDelegateBase.java:250)
        at org.sqlb.call.delegate.server.CallDelegateBase.call(CallDelegateBase.java:63)
        at sun.reflect.GeneratedMethodAccessor2793.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.tx.EjbBMTInterceptor.handleInvocation(EjbBMTInterceptor.java:105)
        at org.jboss.as.ejb3.tx.BMTInterceptor.processInvocation(BMTInterceptor.java:56)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:32)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
        at org.sqlb.call.delegate.server.CallDelegate$$$view4.call(Unknown Source)
        at org.sqlb.call.delegate.client.CallDelegateAUTOWrapper.call(CallDelegateAUTOWrapper.java:54)
        at org.sqlb.call.delegate.client.PreparedCallStub.delegateCall(PreparedCallStub.java:125)
        at bravura.commons.engine.db.DBQueueItem.execPreparedCall(DBQueueItem.java:200)
        at bravura.commons.engine.db.DBQueueItem.execCALL(DBQueueItem.java:281)
        at bravura.commons.engine.db.DBQueueItem.execute(DBQueueItem.java:111)
        at org.sqlb.utils.engine.QueueThread.run(QueueThread.java:163)
      "Transaction Reaper Worker 0":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0000000739b40b88> (a java.util.concurrent.locks.ReentrantLock$FairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:361)
        at org.jboss.jca.adapters.jdbc.local.LocalManagedConnection.rollback(LocalManagedConnection.java:113)
        at org.jboss.jca.core.tx.jbossts.LocalXAResourceImpl.rollback(LocalXAResourceImpl.java:222)
        at com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.rollback(XAOnePhaseResource.java:187)
        at com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord.topLevelAbort(LastResourceRecord.java:126)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2874)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2853)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1616)
        - locked <0x00000006c33457f0> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118)
        at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:217)
        at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370)
        at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)

      Found 1 deadlock.

      More information can be provided on request.

       

      Note that this has been reported in StackOverFlow

      Thank you.