Help with exception - "Could not enlist in transaction on entering meta-aware object..."
We have looked in all we could find about this issue on the internet and could not find something to explain our problem. Sorry for the following long description...
Our application is an EJB 2.1 application initially developed on top of JBOSS 4.2.3 and Oracle 9i DB. In 2008 we moved the application to work in cluster mode and upgraded the DB to Oracle 10G. In 2012 we upgraded the system again to JBOSS 5.1.0, still working in cluster, with Java 1.6. In august 2015 we've upgraded the DB system to Oracle 11G. Since the beginning of 2016 we've been experiencing DB issues. In most cases the issues were related to loosing connections in the DB connection pool. However, on occasion we started seeing the following error:
[org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery.Folder#findByPrimaryKey] Find failed
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a280bcc:dda6:56c51de2:916a4 status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: a280bcc:dda6:56c51de2:916a4 status: ActionStatus.ABORT_ONLY >))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:95)
After a thorough investigation of the lost connections issue we found the problem to be network related, and after changing the DB listening port the problem stopped. However, on occasion we still get the latter error of "could not enlist…"
After looking up this issue on the internet we found a possible solution with adding the following parameter to the configuration:
<property name=”com.arjuna.ats.jta.allowMultipleLastResources” value=”true”/>
After changing this parameter this issue seems to have stopped however, a greater problem surfaced: we get a bunch of the following warning message
2016-04-12 13:03:13,496 WARN [com.arjuna.ats.jta.logging.loggerI18N] (WorkManager(2)-6) [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.multipleWarning] [com.arjuna.ats.internal.jta.transaction.arjunacore.lastResource.multipleWarning] Multiple last resources have been added to the current transaction. This is transactionally unsafe and should not be relied upon. Current resource is org.jboss.resource.connectionmanager.TxConnectionManager$LocalXAResource@1d697ad1
And eventually get an error:
2016-04-12 13:03:31,247 WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkManager(2)-6) [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@35de984c
javax.ejb.EJBException: Update failed. Expected one affected row: rowsAffected=0, id=133105473at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreEntityCommand.execute(JDBCStoreEntityCommand.java:169) at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.storeEntity(JDBCStoreManager.java:666)
Although the error doesn't seem to be related it always appears exactly after the warnings. Changing back the "MultipleLastResources" property to 'false' makes the warning disappear and brings back the enlist issue.
Thinking that this issue is a XAResource issue we decided to move back the application from cluster to single server mode with a LocalTxConnection. This did not solve the problem.
We can now reproduce the error in a specific process. This process does updates to a single DB table using an EJB 2.1 CMP Entity Bean. During this single process we have to insert many objects. The original code opened a UserTransaction, and used batch commits on every 100 objects. The process under the transaction would do a select for info from the DB, some calculations to create the inserted object, sets the results on the entity bean (which translates into commits into the db automatically by the ejb container.)
To further isolate the problem we have taken all selects out of the transaction. We also took out the batch and are now opening the transaction before creating and setting the data on the entity and commits it once all the fields are set. However this doesn't help either. Even though the whole process occurs on a single thread, it fails with the "enlist" issue. Moreover, the failure is on one of the entity set commands, and not the first one (i.e. one or two set commands are successful and then we have the problem).
To sum up. We are now working with JBOSS 5.1.0, Oracle 11G, in single server mode. We're experiencing the "enlist" problem on a single process, running in a single thread, using a "UserTransaction" which is started before creating and setting Data on a single EntityBean and commits at the end of the Data setting.
Could this be database related? database driver related?