In my application i use 20 threads for a job , where each thread calls a remote web service and updates the Oracle Db. This application is deployed in a Jboss server and at every mid night there will be jvm restart. So yesterday while this job was running jboss server got restarted and while restarting i got the error: EJB Invocation failed on component AsynchronousServiceImpl for method public abstract java.util.concurrent.Future com.example.test.hello.rest.service.AsynchronousService.asyncDetails(int,int,int): org.jboss.as.ejb3.component.EJBComponentUnavailableException: JBAS014559: Invocation cannot proceed as component is shutting down.
Once the server got restarted the job was resumed with new 20 threads , but the job got immediately stopped due to Oracle db connecection issue. Error : 2017-11-08 23:36:06,635 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 16,5,main] not responding to interrupt when cancelling TX 0:ffff0a11cb58:-2366ac83:5a038429:6e -- worker marked as zombie and TX scheduled for mark-as-rollback 2017-11-08 23:36:06,638 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012110: TransactionReaper::check successfuly marked TX 0:ffff0a11cb58:-2366ac83:5a038429:6e as rollback only 2017-11-08 23:36:06,638 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 17) ARJUNA012095: Abort of action id 0:ffff0a11cb58:-2366ac83:5a038429:70 invoked while multiple threads active within it. 2017-11-08 23:36:06,638 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:70 in state CANCEL 2017-11-08 23:36:06,638 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 17) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0a11cb58:-2366ac83:5a038429:70 aborting with 1 threads active! 2017-11-08 23:36:06,639 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:72 in state SCHEDULE_CANCEL 2017-11-08 23:36:06,639 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:78 in state SCHEDULE_CANCEL 2017-11-08 23:36:06,639 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:7e in state SCHEDULE_CANCEL 2017-11-08 23:36:07,140 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a11cb58:-2366ac83:5a038429:70 in state CANCEL_INTERRUPTED 2017-11-08 23:36:07,141 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012120: TransactionReaper::check worker Thread[Transaction Reaper Worker 17,5,main] not responding to interrupt when cancelling TX 0:ffff0a11cb58:-2366ac83:5a038429:70 -- worker marked as zombie and TX scheduled for mark-as-rollback
............. and then (EJB default - 16) JBAS014134: EJB Invocation failed on component ServiceDaoImpl for method public abstract long com.example.com.tets.db.daoServiceDao.getNotUpdatedAddressCount(int): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:162) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1] .......... .. Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.7.SP5-redhat-1.jar:4.2.7.SP5-redhat-1]
Are the db connections preoccupied by old threads? Will this threads be destroyed after jvm restart ? This job was working fine before jvm restart, but why this error(connection timeout) occurred after jvm restart and is there any way to fix this issue ? |