1 2 3 Previous Next 32 Replies Latest reply on Oct 2, 2014 2:04 PM by marklittle

    Algorithm for catching and handling TransactionReaper timeouts in EJB methods

    steeltomato

      I am aware of the extensive, related discussion taking place here on how to improve transaction timeout handling JBoss. However, dealing with an immediate need, I am wondering if there is any way currently to return control to the EJB method when a transaction is timed out by the TransactionReaper. This is the simple case where there is just one server (WildFly 9 on the bleeding edge) and one application.

       

      My situation seems like it would be common enough to enterprise applications so I'm surprised that I have found absolutely no way of handling it. I have a data browser page (JSF) that lets users enter complex search criteria to query a database. The query is being executed by an EJB method. I can't control every query situation, so it is relatively easy for a user to run a query that will take significant time to run (hours). The "out-of-the-box" behavior is:

      * User runs long query

      * TransactionReaper attempts to cancel query, fails, marks as rollback only

      * EJB continues waiting on query method that will never complete, DB connection stays open, query continues running

      * User gets frustrated and reloads the page, tries again

      * All DB connections become engaged in long-running queries

      * Server effectively hangs

       

      I absolutely do not want to change the timeout. Timing out these queries is the correct behavior, otherwise the DB will eventually be overwhelmed and the application will choke. I need a way to actively kill these long queries so that control is returned and the user can be notified that their query took too long to run.

       

      Is there currently any way to cleanly handle this situation?

       

      I don't think it's necessary since it is expected behavior, but for reference here is the trace during cancellation:

      15:21:18,778 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a258102:-6726da35:5429af3b:5ae in state  RUN
      15:21:18,779 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff0a258102:-6726da35:5429af3b:5ae invoked while multiple threads active within it.
      15:21:18,780 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0a258102:-6726da35:5429af3b:5ae aborting with 1 threads active!
      15:21:19,279 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a258102:-6726da35:5429af3b:5ae in state  CANCEL
      15:21:19,280 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012378: ReaperElement appears to be wedged: sun.misc.Unsafe.park(Native Method)
      java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
      java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
      java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229)
      java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
      org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:374)
      org.jboss.jca.adapters.jdbc.local.LocalManagedConnection.rollback(LocalManagedConnection.java:113)
      org.jboss.jca.core.tx.jbossts.LocalXAResourceImpl.rollback(LocalXAResourceImpl.java:242)
      com.arjuna.ats.internal.jta.resources.arjunacore.XAOnePhaseResource.rollback(XAOnePhaseResource.java:205)
      com.arjuna.ats.internal.arjuna.abstractrecords.LastResourceRecord.topLevelAbort(LastResourceRecord.java:126)
      com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2980)
      com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2959)
      com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1666)
      com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:127)
      com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
      com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:381)
      com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)
      
      
      15:21:19,782 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a258102:-6726da35:5429af3b:5ae in state  CANCEL_INTERRUPTED
      15:21:19,782 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) RJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 0,5,main] not responding to interrupt when cancelling TX 0:ffff0a258102:-6726da35:5429af3b:5ae -- worker marked as zombie and TX scheduled for mark-as-rollback
      15:21:19,783 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffff0a258102:-6726da35:5429af3b:5ae as rollback only
      
      
      
      
        1 2 3 Previous Next