13 Replies Latest reply on Dec 7, 2008 9:38 PM by Mohit Anchlia

    Perplexing error  - meta-aware object error

    Mohit Anchlia Novice

      I am seeing following error: could someone help:

      2008-11-20 17:30:05,342 ERROR [hibernate.util.JDBCExceptionReporter] - Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))

      javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
      at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
      at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
      at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)

        • 1. Re: Perplexing error  - meta-aware object error
          Peter Johnson Master

          Looks like the app server was unable to a connect to the database. Please post the full stack trace and the *-ds.xml file for your data source.

          Remember to enclose your XML text in UBBCode "code" tags - you can do this by selecting the XML text and clicking the Code button above the editor window. Also, click the Preview button to ensure that the formatting is correct and the XML text shows up before posting.

          • 2. Re: Perplexing error  - meta-aware object error
            Mohit Anchlia Novice

            Pool size is set to 250. And I have been monitoring DS in JMX and always see atleast 150 free connections. It's really random. I have looked at oracle and I see just 250 sessions overall at any given point of time. It's really perplexing.


            <local-tx-datasource>
            <jndi-name>EDS</jndi-name>
            <connection-url>jdbc:oracle:thin:@(description = (address_list = (address = (protocol = tcp)(host = e3.p.i.com)(port = 1526)))(connect_data = (sid = eper3)))</connection-url>
            <connection-property name="SetBigStringTryClob">true</connection-property>
            <!-- recommended by Oracle support to avoid ORA-01461 errors -->
            <connection-property name="oracle.jdbc.RetainV9LongBindBehavior">true</connection-property>
            <max-pool-size>250</max-pool-size>
            <driver-class>oracle.jdbc.OracleDriver</driver-class>
            <!-- Uses the pingDatabase method to check a connection is still valid before handing it out from the pool -->
            <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
            <!-- Checks the Oracle error codes and messages for fatal errors -->
            <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
            <!-- sql to call when connection is created
            <new-connection-sql>some arbitrary sql</new-connection-sql>
            -->

            <!-- sql to call on an existing pooled connection when it is obtained from pool - the OracleValidConnectionChecker is prefered
            <check-valid-connection-sql>some arbitrary sql</check-valid-connection-sql>
            -->

            <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->

            <type-mapping>Oracle9i</type-mapping>


            <security-domain>EncryptedOracleDbRealm</security-domain>
            </local-tx-datasource>



            -----
            c.i.e.ef.ee.ag.core.exceptions.AgencyException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
            at c.i.e.ef.ee.ag.core.AckReceiver.createAIterator(AckReceiver.java:88)
            at c.i.e.ef.ee.ag.core.AckReceiver.receiveAs(AckReceiver.java:49)
            at c.i.e.ef.ee.ag.core.RegulatorAppService.receiveAs(RegulatorAppService.java:97)
            at c.i.e.ef.ee.ag.core.RegulatorBean.ejbTimeout(RegulatorBean.java:75)
            at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
            at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
            at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
            at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
            at org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
            at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
            at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
            at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
            at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
            at org.jboss.ejb.Container.invoke(Container.java:960)
            at org.jboss.ejb.txtimer.TimedObjectInvokerImpl.callTimeout(TimedObjectInvokerImpl.java:99)
            at org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
            at java.util.TimerThread.mainLoop(Timer.java:512)
            at java.util.TimerThread.run(Timer.java:462)
            Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
            at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
            at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
            at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)
            at c.i.e.ef.ee.dao.em.EmRegulatorCfgDao.getCfg(EmRegulatorCfgDao.java:24)
            at c.i.e.ef.ee.dao.em.EmRegulatorCfgDao.getValidCfg(EmRegulatorCfgDao.java:15)
            at c.i.e.ef.ee.ag.core.AgencyUtils.getRegulatorCfg(AgencyUtils.java:44)
            at c.i.e.ef.ee.ag.core.AckReceiver.createAckIterator(AckReceiver.java:80)
            ... 23 more
            Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
            at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
            at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)

            at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
            at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
            at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
            at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
            at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
            at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
            at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
            at org.hibernate.loader.Loader.doQuery(Loader.java:673)
            at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
            at org.hibernate.loader.Loader.loadEntity(Loader.java:1860)
            at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
            at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
            at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3044)
            at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395)
            at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
            at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
            at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
            at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
            at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
            at org.hibernate.impl.SessionImpl.get(SessionImpl.java:815)
            at org.hibernate.impl.SessionImpl.get(SessionImpl.java:808)
            at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:174)

            ... 28 more
            Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
            at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
            at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
            ... 46 more
            Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!)
            at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
            ... 48 more
            Caused by: java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!
            at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:525)
            at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
            ... 52 more

            • 3. Re: Perplexing error  - meta-aware object error
              Mohit Anchlia Novice

              Could someone help with this one. This totally random and system is not under any load. I've checked current connections in Oracle, checked sockets etc. but nothing seems to be standing out. Is there a way to dig deeper.

              • 4. Re: Perplexing error  - meta-aware object error
                Mohit Anchlia Novice

                No one has replied. I still these errors and everything on DB, active sessions, sockets, connection look fine.

                • 5. Re: Perplexing error  - meta-aware object error
                  Mohit Anchlia Novice

                  We are getting these errors a lot now and there are no issues on Oracle side. It works fine under low load but when we apply high load we start to see these errors. I am wondering if there is any problem with Hibernate connection pooling. May be connections are closed but are still on the pool? Could someone try and help.

                  • 6. Re: Perplexing error  - meta-aware object error
                    Mohit Anchlia Novice

                    I enabled hibernate trace and see the following, it looks like some state is not getting clear and it thinks that the connection thread is already in running state. Could someone help?

                    Here is the additional info in addition to the stack trace that I earlier posted

                    Caused by: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!))
                     at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:94)
                     at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
                     at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
                     ... 49 more
                    Caused by: org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!)
                     at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:343)
                     at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:518)
                     at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:399)
                     at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842)
                     at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88)
                     ... 51 more
                    Caused by: java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a5174:985a:493439f5:4c5a0 status: ActionStatus.RUNNING > in enlist!
                     at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:525)
                     at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:337)
                     ... 55 more


                    • 7. Re: Perplexing error  - meta-aware object error
                      jaikiran pai Master

                      I don't have much idea about what might be wrong, but are you sure the transaction is not timing out or maybe you are using multiple non-XA transaction aware resources in a single transaction? For the latter, see this http://www.jboss.org/community/docs/DOC-11443

                      • 8. Re: Perplexing error  - meta-aware object error
                        ivan Newbie

                        I had a similar problem moving EJB 2.0 ear from
                        JBoss 4.0.5 to JBoss 4.2.2.GA.

                        Look for
                        in conf/jbossjta-properties.xml and add the property


                        The doc that jaikiran mensions has some info about this property.

                        eg:




                        <!-- ... other properties ... -->

                        • 9. Re: Perplexing error  - meta-aware object error
                          ivan Newbie

                          The XML did not copy!
                          put

                          <property name="com.arjuna.ats.jta.allowMultipleLastResources" value="true"/>


                          in

                          <properties depends="arjuna" name="jta">
                           <property name="com.arjuna.ats.jta.allowMultipleLastResources" value="true"/>
                          
                          
                          </properties>


                          • 10. Re: Perplexing error  - meta-aware object error
                            Mohit Anchlia Novice

                            But I don't have multiple local resources. All I have is JMS and Oracle. Oracle is local resource and JMS is XA so I would guess it should be using last resource gambit. Isn't that correct? Other than that all we have is Session Beans (that might call another bean with requiresNew) and MDB. Why would this need to have allowMultipleLastResource. Besides error message suggests something else:

                            2008-11-20 17:30:05,342 ERROR [hibernate.util.JDBCExceptionReporter] - Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Trying to change transaction TransactionImple < ac, BasicAction: a0a515c:cd7e:4925f02a:42a581 status: ActionStatus.RUNNING > in enlist!))
                            
                            javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection
                            at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
                            at org.hibernate.ejb.AbstractEntityManagerImpl.find(AbstractEntityManagerImpl.java:194)
                            at org.jboss.ejb3.entity.TransactionScopedEntityManager.find(TransactionScopedEntityManager.java:171)


                            Am I missing any point here?

                            • 11. Re: Perplexing error  - meta-aware object error
                              Mohit Anchlia Novice

                              We have one local resource and one XA resource. Why would we still need multiple last resources enabled?

                              • 12. Re: Perplexing error  - meta-aware object error
                                jaikiran pai Master

                                 

                                "mohitanchlia" wrote:
                                We have one local resource and one XA resource.


                                So if both these resources are part of the same transaction, there is no way the local datasource resource can follow the two phase commit and hence the exception. If you want the datasource to be part of a transaction which has more than one transactional resource involved, then you need to configure that as XA.

                                • 13. Re: Perplexing error  - meta-aware object error
                                  Mohit Anchlia Novice

                                  As I mentioned that we have one local resource and one XA resource. And it looks like Jboss uses last resource gambit in this scenario. Also note that this error occurs in high load.