Sequence of warn/errors in production java application
our old application developed with the technologies Hibernate 4.2, EJB 3, struts2.5.13. and application server is JBOSS openshift thisapplication will insert the data to database and also have a job that runs for every 1 hour.
for example, the job starts at 10 am to send the data from xxx table to another module. at the same time 10 am the user can insert the data to xxx table. so the possibility of concurrently accessing the same row by two threads (1 is job thread and another one is application thread) is there.
We got the following sequence of errors in production.
11:33:33,026 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff0a012d75:-5e79fb5e:5a83bd7e:1167 aborting with 1 threads active!
11.37.26.229 Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key Htg7hYujo1IihaXKvJwxalAg1fOBSppgm5HI8wEA and requestor GlobalTransaction
11.37.26.229 Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key Htg7hYujo1IihaXKvJwxalAg1fOBSppgm5HI8wEA and requestor GlobalTransaction:<pod-name>:1762:local. Lock is held by GlobalTransaction:<pod-name>:1641:local
11:37:41,232 org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key Htg7hYujo1IihaXKvJwxalAg1fOBSppgm5HI8wEA and requestor GlobalTransaction:<pod-name>:1762:local. Lock is held by GlobalTransaction:<pod-name>:1641:local
11.42.31 Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key Htg7hYujo1IihaXKvJwxalAg1fOBSppgm5HI8wEA and requestor GlobalTransaction:<pod-name>:1776:local. Lock is held by GlobalTransaction:<pod-name>:1641:local
2018-02-14 11:44:02.454 | pod-name | at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConnection.java:6295) |
2018-02-14 11:44:05.242 | pod-name-17-jmp0l | [0m2018-02-14 11:44:05,242 ERROR unexpected error javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not prepare statement |
I am thinking that these errors are because of the threads/transactions concurrently trying to access the same row in xxx table. And the data source in standalone config file doesn't have the validation check on the connection pool objects. So, i am suggesting the solution to change the job timing and keep validation for connection pool.
I am new to JBOSS. I would like to verify i am on right track.
Recently moved this application from glassfish to jboss - openshift.
Any suggestions are welcome. Please help me with this.
Thanks