Perplexing error - meta-aware object error

[mohitanchlia]

I am seeing following error: could someone help:

2008-11-20 17:30:05,342 ERROR [hibernate.util.JDBCExceptionReporter] - Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))

javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)

[peterj]

Looks like the app server was unable to a connect to the database. Please post the full stack trace and the *-ds.xml file for your data source.

Remember to enclose your XML text in UBBCode "code" tags - you can do this by selecting the XML text and clicking the Code button above the editor window. Also, click the Preview button to ensure that the formatting is correct and the XML text shows up before posting.

[mohitanchlia]

Pool size is set to 250. And I have been monitoring DS in JMX and always see atleast 150 free connections. It's really random. I have looked at oracle and I see just 250 sessions overall at any given point of time. It's really perplexing.


<local-tx-datasource>
<jndi-name>EDS</jndi-name>
<connection-url>jdbc:oracle:thin:@(description = (address_list = (address = (protocol = tcp)(host = e3.p.i.com)(port = 1526)))(connect_data = (sid = eper3)))</connection-url>
<connection-property name="SetBigStringTryClob">true</connection-property>
<!-- recommended by Oracle support to avoid ORA-01461 errors -->
<connection-property name="oracle.jdbc.RetainV9LongBindBehavior">true</connection-property>
<max-pool-size>250</max-pool-size>
<driver-class>oracle.jdbc.OracleDriver</driver-class>
<!-- Uses the pingDatabase method to check a connection is still valid before handing it out from the pool -->
<valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
<!-- Checks the Oracle error codes and messages for fatal errors -->
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<!-- sql to call when connection is created
<new-connection-sql>some arbitrary sql</new-connection-sql>
-->

<!-- sql to call on an existing pooled connection when it is obtained from pool - the OracleValidConnectionChecker is prefered
<check-valid-connection-sql>some arbitrary sql</check-valid-connection-sql>
-->

<!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->

<type-mapping>Oracle9i</type-mapping>


<security-domain>EncryptedOracleDbRealm</security-domain>
</local-tx-datasource>



-----
c.i.e.ef.ee.ag.core.exceptions.AgencyException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at c.i.e.ef.ee.ag.core.AckReceiver.createAIterator(AckReceiver.java:88)
at c.i.e.ef.ee.ag.core.AckReceiver.receiveAs(AckReceiver.java:49)
at c.i.e.ef.ee.ag.core.RegulatorAppService.receiveAs(RegulatorAppService.java:97)
at c.i.e.ef.ee.ag.core.RegulatorBean.ejbTimeout(RegulatorBean.java:75)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.txtimer.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:99)
at org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)
at c.i.e.ef.ee.dao.em.EmRegulatorCfgDao.getCfg(EmRegulatorCfgDao.java:24)
at c.i.e.ef.ee.dao.em.EmRegulatorCfgDao.getValidCfg(EmRegulatorCfgDao.java:15)
at c.i.e.ef.ee.ag.core.AgencyUtils.getRegulatorCfg(AgencyUtils.java:44)
at c.i.e.ef.ee.ag.core.AckReceiver.createAckIterator(AckReceiver.java:80)
... 23 more
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)

at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
at org.hibernate.loader.Loader.doQuery(Loader.java:673)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1860)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3044)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:174)

... 28 more
Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
... 46 more
Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
... 48 more
Caused by: java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:525)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
... 52 more

[mohitanchlia]

Could someone help with this one. This totally random and system is not under any load. I've checked current connections in Oracle, checked sockets etc. but nothing seems to be standing out. Is there a way to dig deeper.

[mohitanchlia]

No one has replied. I still these errors and everything on DB, active sessions, sockets, connection look fine.

[mohitanchlia]

We are getting these errors a lot now and there are no issues on Oracle side. It works fine under low load but when we apply high load we start to see these errors. I am wondering if there is any problem with Hibernate connection pooling. May be connections are closed but are still on the pool? Could someone try and help.

[mohitanchlia]

I enabled hibernate trace and see the following, it looks like some state is not getting clear and it thinks that the connection thread is already in running state. Could someone help?

Here is the additional info in addition to the stack trace that I earlier posted

Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!))
 at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
 at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
 at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
 ... 49 more
Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!)
 at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
 at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
 at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
 ... 51 more
Caused by: java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!
 at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:525)
 at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
 ... 55 more

[jaikiran]

I don't have much idea about what might be wrong, but are you sure the transaction is not timing out or maybe you are using multiple non-XA transaction aware resources in a single transaction? For the latter, see this http://www.jboss.org/community/docs/DOC-11443

[ivanlc]

I had a similar problem moving EJB 2.0 ear from
JBoss 4.0.5 to JBoss 4.2.2.GA.

Look for
in conf/jbossjta-properties.xml and add the property


The doc that jaikiran mensions has some info about this property.

eg:




<!-- ... other properties ... -->

[ivanlc]

The XML did not copy!
put

<property name="com.arjuna.ats.jta.allowMultipleLastResources" value="true"/>

in

<properties depends="arjuna" name="jta">
 <property name="com.arjuna.ats.jta.allowMultipleLastResources" value="true"/>


</properties>

[mohitanchlia]

But I don't have multiple local resources. All I have is JMS and Oracle. Oracle is local resource and JMS is XA so I would guess it should be using last resource gambit. Isn't that correct? Other than that all we have is Session Beans (that might call another bean with requiresNew) and MDB. Why would this need to have allowMultipleLastResource. Besides error message suggests something else:

2008-11-20 17:30:05,342 ERROR [hibernate.util.JDBCExceptionReporter] - Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))

javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)

Am I missing any point here?

[mohitanchlia]

We have one local resource and one XA resource. Why would we still need multiple last resources enabled?

[jaikiran]

"mohitanchlia" wrote:
We have one local resource and one XA resource.

So if both these resources are part of the same transaction, there is no way the local datasource resource can follow the two phase commit and hence the exception. If you want the datasource to be part of a transaction which has more than one transactional resource involved, then you need to configure that as XA.

[mohitanchlia]

As I mentioned that we have one local resource and one XA resource. And it looks like Jboss uses last resource gambit in this scenario. Also note that this error occurs in high load.