Deadlock in JCA during JDBC pool flush
Hi
I am using Jboss-EAP-6.1 with ironjacamar version 1.0.26. During an outage of the database application stopped responding. Thread dump shows that there is a in the jca layer. We use autoreconnect and <flush-strategy>EntirePool</flush-strategy>. Below is the stack trace of the offending threads. Is this a know issue fixed in later version ?
Found one Java-level deadlock:
=============================
"http-/0.0.0.0:8080-6":
waiting to lock monitor 0x0000000001e64678 (object 0x0000000781621fd0, a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject),
which is held by "ajp-xxx/10.155.78.226:8009-34"
"ajp-xxx/10.155.78.226:8009-34":
waiting to lock monitor 0x00007f7eb4004a88 (object 0x000000079ff923e8, a com.mysql.jdbc.jdbc2.optional.JDBC4MysqlXAConnection),
which is held by "Thread-1006 (HornetQ-client-global-threads-1821340891)"
"Thread-1006 (HornetQ-client-global-threads-1821340891)":
waiting to lock monitor 0x0000000001e64678 (object 0x0000000781621fd0, a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject),
which is held by "ajp-xxx/10.155.78.226:8009-34"
==================================
"http-/0.0.0.0:8080-6":
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.initLock(AbstractPool.java:235)
- waiting to lock <0x0000000781621fd0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getLock(AbstractPool.java:267)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionOldConnection(AbstractPool.java:443)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:369)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
"Thread-1006 (HornetQ-client-global-threads-1821340891)":
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.flush(AbstractPool.java:315)
- waiting to lock <0x0000000781621fd0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject)
at org.jboss.jca.core.connectionmanager.listener.AbstractConnectionListener.connectionErrorOccurred(AbstractConnectionListener.java:367)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.connectionErrorOccurred(TxConnectionListener.java:445)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:647)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.broadcastConnectionError(XAManagedConnection.java:199)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection$1.connectionErrorOccurred(XAManagedConnection.java:87)
at com.mysql.jdbc.jdbc2.optional.MysqlPooledConnection.callConnectionEventListeners(MysqlPooledConnection.java:248)
- locked <0x000000079ff923e8> (a com.mysql.jdbc.jdbc2.optional.JDBC4MysqlXAConnection)
at com.mysql.jdbc.jdbc2.optional.WrapperBase.checkAndFireConnectionError(WrapperBase.java:61)
at com.mysql.jdbc.jdbc2.optional.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:858)
at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:107)
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:462)
"ajp-xxx/10.155.78.226:8009-34":
at com.mysql.jdbc.jdbc2.optional.ConnectionWrapper.close(ConnectionWrapper.java:830)
- waiting to lock <0x000000079ff923e8> (a com.mysql.jdbc.jdbc2.optional.JDBC4MysqlXAConnection)
at com.mysql.jdbc.jdbc2.optional.ConnectionWrapper.close(ConnectionWrapper.java:466)
at com.mysql.jdbc.jdbc2.optional.JDBC4ConnectionWrapper.close(JDBC4ConnectionWrapper.java:72)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.destroy(BaseWrapperManagedConnection.java:494)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.destroy(XAManagedConnection.java:225)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.doDestroy(SemaphoreArrayListManagedConnectionPool.java:819)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.flush(SemaphoreArrayListManagedConnectionPool.java:568)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.flush(AbstractPool.java:323)
- locked <0x0000000781621fd0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject)
at org.jboss.jca.core.connectionmanager.listener.AbstractConnectionListener.connectionErrorOccurred(AbstractConnectionListener.java:367)
at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.connectionErrorOccurred(TxConnectionListener.java:445)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:647)
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:522)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:497)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:284)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:495)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:374)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)