Database related thread is stuck - lockName:java.util.concurrent.locks.ReentrantLock$NonfairSync@5bf2136d
I am using JBoss AS 4.2.3 with JDK 6. A production issue manifested itself a few times and an important thread that does some background processing got stuck. The entity manager is used from within a stateless session bean to find a specific JPA entity:
...
_em.find(MerchantLocation.class, locationId);
...
Looking at the thread dump, it seems like a certain java.util.concurrent.locks.ReentrantLock has the lock method invoked but it gets stuck there. I am attaching the source code for BaseWrapperManagedConnection.java. Nothing jumps at me apart from the following at the cleanup method, which I can't really think of what to make of it:
...
// I'm recreating the lock object when we return to the pool
// because it looks too nasty to expect the connection handle
// to unlock properly in certain race conditions
// where the dissociation of the managed connection is "random".
lock = new ReentrantLock();
...
Here is the relevant stack trace (taken from thread dump):
Thread: Queue Manager : priority:1, demon:true, threadId:214, threadState:WAITING, lockName:java.util.concurrent.locks.ReentrantLock$NonfairSync@5bf2136d
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:254)
org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:111)
org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.start(JcaXAResourceWrapper.java:113)
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:788)
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:499)
org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:788)
org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:579)
org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:352)
org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:524)
org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:405)
org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:849)
org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
org.hibernate.loader.Loader.doQuery(Loader.java:673)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
org.hibernate.loader.Loader.loadEntity(Loader.java:1860)
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3049)
org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:399)
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846)
org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557)
org.hibernate.type.EntityType.resolve(EntityType.java:379)
org.hibernate.type.OneToOneType.assemble(OneToOneType.java:141)
org.hibernate.type.TypeFactory.assemble(TypeFactory.java:420)
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:96)
org.hibernate.cache.entry.CacheEntry.assemble(CacheEntry.java:82)
org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:557)
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:512)
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:357)
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:182)
org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)
Has anyone came across a similar issue like this?