In a WildFly 8.2.1-Final high-volume installation we are experiencing degrading performance within about a week until restart is necessary along with more and more frequently occurring failures to send JMS messages because of transactions in state ABORT_ONLY:
Caused by: javax.ejb.EJBException: javax.jms.JMSException: Could not create a session: IJ000457: Unchecked throwable in managedConnectionReconnected()
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@3915e409[state=NORMAL managed connection=org.hornetq.ra.HornetQRAManagedConnection@7d6bcd06 connection handles=0 lastUse=1494579251303 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri@691085ec mcp=SemaphoreArrayListManagedConnectionPool@66a127f5[pool=HornetQConnectionDefinition] xaResource=XAResourceWrapperImpl@56a9bb4[xaResource=org.hornetq.ra.HornetQRAXAResource@4ca920eb pad=false overrideRmValue=null productName=HornetQ productVersion=2.0 jndiName=java:/JmsXA] txSync=null]
... 261 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
... 263 more
Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff0a0001c3:682851c6:5900737b:dd09939 status: ActionStatus.ABORT_ONLY >
... 264 more
We are sending the messages from an EJB with CMT using the pooled connection factory with transaction mode XA and there is also an Oracle XA database connection involved in the transaction. We are taking care to close all resources such as the connection, the session and the producer in a finally block including null checks and it does not look like we have a resource leak.
Also, for the time being, we have no idea why the transaction has status ABORT_ONLY. We don't set it to rollback only and, at least not before the error occurs and in the same thread, there is nothing in the log about an EJBTransactionRolledbackException or any other exception being thrown that would cause the transaction to be rolled back or marked for rollback only.
While the error occurred thousands of times in the log, it seems there are all in all only 7 instances of
Is it possible that over time, more and more connections somehow "break" in regards to the transaction and keep on getting handed out by the pool? This maybe fits the fact that the error occurs more and more frequently over time.
If yes, is there a way to purge one or all connections from the pool so it fills itself with "brand new" connections?
We have tried hard to reproduce the error in many ways without success so far. By marking the transaction for rollback only before trying to send the message, we get a somewhat similar error in the log, but it happens in
TxConnectionManagerImpl.getManagedConnection, not in
TxConnectionManagerImpl.managedConnectionReconnected as in the real error, which maybe is a significant difference.