2 Replies Latest reply on Nov 10, 2017 8:54 PM by jobyywilson

    JBoss arjuna TransactionReaper timeout after jvm restart

    jobyywilson
    jobyywilson Nov 9, 2017 9:21 AM

      0down votefavorite

       

       

      In my application i use 20 threads for a job , where each thread calls a remote web service and updates the Oracle Db. This application is deployed in a Jboss server and at every mid night there will be jvm restart. So yesterday while this job was running jboss server got restarted and while restarting i got the error:

      EJB Invocation failed on component AsynchronousServiceImpl for method public abstract java.util.concurrent.Future com.example.test.hello.rest.service.AsynchronousService.asyncDetails(int,int,int): org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down.

      Once the server got restarted the job was resumed with new 20 threads , but the job got immediately stopped due to Oracle db connecection issue. Error :

      2017-11-08 23:36:06,635 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 16,5,main] not responding to interrupt when cancelling TX 0:ffff0a11cb58:-2366ac83:5a038429:6e -- worker marked as zombie and TX scheduled for mark-as-rollback
      2017-11-08 23:36:06,638 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffff0a11cb58:-2366ac83:5a038429:6e as rollback only
      2017-11-08 23:36:06,638 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 17) ARJUNA012095: Abort of action id 0:ffff0a11cb58:-2366ac83:5a038429:70 invoked while multiple threads active within it.
      2017-11-08 23:36:06,638 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:70 in state CANCEL
      2017-11-08 23:36:06,638 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 17) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0a11cb58:-2366ac83:5a038429:70 aborting with 1 threads active!
      2017-11-08 23:36:06,639 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:72 in state SCHEDULE_CANCEL
      2017-11-08 23:36:06,639 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:78 in state SCHEDULE_CANCEL
      2017-11-08 23:36:06,639 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:7e in state SCHEDULE_CANCEL
      2017-11-08 23:36:07,140 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:70 in state CANCEL_INTERRUPTED
      2017-11-08 23:36:07,141 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 17,5,main] not responding to interrupt when cancelling TX 0:ffff0a11cb58:-2366ac83:5a038429:70 -- worker marked as zombie and TX scheduled for mark-as-rollback

       

      ............. and then

       (EJB default - 16) JBAS014134: EJB Invocation failed on component ServiceDaoImpl for method public abstract long com.example.com.tets.db.daoServiceDao.getNotUpdatedAddressCount(int): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:162) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]  .......... ..  Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.7.SP5-redhat-1.jar:4.2.7.SP5-redhat-1]

      Are the db connections preoccupied by old threads?

      Will this threads be destroyed after jvm restart ?

      This job was working fine before jvm restart, but why this error(connection timeout) occurred after jvm restart and is there any way to fix this issue ?

        • 1. Re: JBoss arjuna TransactionReaper timeout after jvm restart
          andey
          andey Nov 10, 2017 6:46 AM (in response to jobyywilson)

          - Long running transactions may have several possible causes include performance issues (on the host running JBoss, in the network or on the database server), application issues such as poorly restricted queries, deadlock, etc. The root cause of long running transactions is usually not addressed by increasing the timeout period. Instead, increasing the time the transaction is allowed to run without any attempt to interrupt it usually makes the impact of one or more problematic transactions worse on the entire system.

           

          - transaction may need attention if it runs for two minutes or more. The default timeout period of 5 minutes (300 seconds) is meant to catch problematic transactions and attempt to minimize their impact by attempting to free database resources, marking the transaction as ABORT_ONLY to produce quick failure for subsequent operations allowing application code to cleanup and rollback, etc. Increasing the timeout period to 15 minutes may negatively impact the entire system.

           

          Recommend increasing transaction timeout periods in most cases. Investigate application source code to determine why the transaction is taking so much long time as a result it was timed out.

            • Actions
          • 2. Re: JBoss arjuna TransactionReaper timeout after jvm restart
            jobyywilson
            jobyywilson Nov 10, 2017 8:54 PM (in response to andey)

            so is this timeout occurred due to all the connections in the connection pool are busy ?

              • Actions