3 Replies Latest reply on Nov 13, 2010 7:15 AM by Wolf-Dieter Fink

    oracle.jdbc.xa.OracleXAException during jboss clustering

    Bala Tubati Newbie

      Hi,

       

      Our application is working as expected with single instance.  After configuring the clustering we are observing the Oracle XA error message.

       

      Software Used:

      Java Version: 1.5.22(Java HotSpot(TM) 64-Bit Server VM )
      JBoss Version: 4.0.2
      Database Product Version : Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
      JDBC Driver : Oracle JDBC driver 10.2.0.4.0
      MQ Server: 6.0.2
      Apache mod jk load balancer: httpd-2.2.16

       

      Steps:


      1. Shutdown both the nodes.
      2. Make sure no messages exist in the queue.
      3. Start the node 1 and make sure the node is started.
      4. Start the node 2.
      5. Run the test program to post 1000 message in queue.
      6. Observe the server log of the both the nodes.

       

      Actual result:

      Both the nodes are showing Oracle XA Error. .  This behavior is always reproducible.
      16:05:51,250 WARN  [TransactionImpl] XAException: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=t710/396, BranchQual=, localId=396] errorCode=XAER_RMERR

      [java] oracle.jdbc.xa.OracleXAException
      [java]  at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1120)
      [java]  at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:249)
      [java]  at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:131)
      [java]  at org.jboss.tm.TransactionImpl$Resource.startResource(TransactionImpl.java:1952)
      [java]  at org.jboss.tm.TransactionImpl.enlistResource(TransactionImpl.java:548)
      [java]  at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:436)
      [java]  at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322)
      [java]  at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:583)
      [java]  at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450)
      [java]  at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
      [java]  at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
      [java]  at oracle.toplink.jndi.JNDIConnector.connect(JNDIConnector.java:110)
      [java]  at oracle.toplink.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:147)
      [java]  at oracle.toplink.internal.databaseaccess.DatasourceAccessor.connect(DatasourceAccessor.java:197)
      [java]  at oracle.toplink.internal.databaseaccess.DatabaseAccessor.connect(DatabaseAccessor.java:220)
      [java]  at oracle.toplink.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:400)
      [java]  at oracle.toplink.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1223)
      [java]  at oracle.toplink.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:184)
      [java]  at oracle.toplink.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:485)
      [java]  at oracle.toplink.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:441)
      [java]  at oracle.toplink.threetier.ServerSession.executeCall(ServerSession.java:457)
      [java]  at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:117)
      [java]  at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:103)
      [java]  at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:501)
      [java]  at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:862)
      [java]  at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:837)
      [java]  at oracle.toplink.queryframework.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:415)
      [java]  at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:811)
      [java]  at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:620)
      [java]  at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
      [java]  at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
      [java]  at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2073)
      [java]  at oracle.toplink.publicinterface.Session.executeQuery(Session.java:988)
      [java]  at oracle.toplink.publicinterface.Session.executeQuery(Session.java:945)
      [java]  at oracle.toplink.publicinterface.Session.readObject(Session.java:2588)
      [java]  at com.perotsystems.diamond.tools.toplink.ToplinkPersistenceWrapper.load(ToplinkPersistenceWrapper.java:629)
      [java]  at com.perotsystems.diamond.tools.toplink.ToplinkPersistenceWrapper.load(ToplinkPersistenceWrapper.java:597)
      [java]  at com.perotsystems.diamond.persistence.service.acl.adapter.diamond950.toplink.Diamond950AclPersistenceAdapterImpl.readAcl(Diamond950AclPersistenceAdapterImpl.java:49)
      [java]  at com.perotsystems.diamond.persistence.service.acl.AclPersistenceServiceImpl.readObject(AclPersistenceServiceImpl.java:101)
      [java]  at com.perotsystems.diamond.persistence.PersistenceServiceImpl.readObject(PersistenceServiceImpl.java:356)
      [java]  at com.perotsystems.diamond.persistence.PersistenceServiceImpl.readObject(PersistenceServiceImpl.java:379)
      [java]  at com.perotsystems.diamond.security.SecurityServiceImpl.loadAcl(SecurityServiceImpl.java:867)
      [java]  at com.perotsystems.diamond.security.SecurityServiceImpl.login(SecurityServiceImpl.java:324)
      [java]  at com.perotsystems.diamond.security.SecurityServiceImpl.login(SecurityServiceImpl.java:226)
      [java]  at com.perotsystems.diamond.security.SecurityServiceSessionBean.login(SecurityServiceSessionBean.java:141)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      [java]  at java.lang.reflect.Method.invoke(Method.java:592)
      [java]  at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
      [java]  at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
      [java]  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
      [java]  at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:130)
      [java]  at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:51)
      [java]  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
      [java]  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
      [java]  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
      [java]  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
      [java]  at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
      [java]  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
      [java]  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
      [java]  at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
      [java]  at org.jboss.ejb.Container.invoke(Container.java:873)
      [java]  at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:413)
      [java]  at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
      [java]  at $Proxy786.login(Unknown Source)
      [java]  at com.perotsystems.diamond.security.clientproxy.SecurityServiceClientProxyEJBLocalImpl.login(SecurityServiceClientProxyEJBLocalImpl.java:106)
      [java]  at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:89)
      [java]  at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:112)
      [java]  at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:101)
      [java]  at com.perotsystems.diamond.DiamondWebService.login(DiamondWebService.java:275)
      [java]  at com.perotsystems.diamond.DiamondWebService.getSOAPSecureSession(DiamondWebService.java:222)
      [java]  at com.perotsystems.diamond.DiamondWebService.getSecureSession(DiamondWebService.java:316)
      [java]  at com.perotsystems.diamond.ddsservice.ws.DdsServiceSoapBindingImpl.processBatchDirect(DdsServiceSoapBindingImpl.java:592)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      [java]  at java.lang.reflect.Method.invoke(Method.java:592)
      [java]  at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
      [java]  at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
      [java]  at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
      [java]  at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
      [java]  at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
      [java]  at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
      [java]  at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
      [java]  at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
      [java]  at com.perotsystems.diamond.framework.mdb.ws.WebServiceUtil.invoke(WebServiceUtil.java:107)
      [java]  at com.perotsystems.diamond.framework.mdb.ws.DispatchProcessor.invokeWebService(DispatchProcessor.java:64)
      [java]  at com.perotsystems.diamond.framework.mdb.ws.DispatchProcessor.processMessage(DispatchProcessor.java:53)
      [java]  at com.perotsystems.diamond.framework.mdb.basic.BasicMessageController.processMessage(BasicMessageController.java:66)
      [java]  at com.perotsystems.diamond.framework.mdb.DiamondMessageDrivenBeanImpl.processMessage(DiamondMessageDrivenBeanImpl.java:208)
      [java]  at com.perotsystems.diamond.framework.mdb.DiamondMessageDrivenBeanImpl.onMessage(DiamondMessageDrivenBeanImpl.java:175)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      [java]  at java.lang.reflect.Method.invoke(Method.java:592)
      [java]  at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
      [java]  at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:475)
      [java]  at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
      [java]  at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:87)
      [java]  at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
      [java]  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
      [java]  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
      [java]  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
      [java]  at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94)
      [java]  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
      [java]  at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
      [java]  at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389)
      [java]  at org.jboss.ejb.Container.invoke(Container.java:873)
      [java]  at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1090)
      [java]  at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
      [java]  at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256)
      [java]  at com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:147)
      [java]  at com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2634)
      [java]  at com.ibm.mq.jms.MQSession.run(MQSession.java:862)
      [java]  at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:182)
      [java]  at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
      [java]  at java.lang.Thread.run(Thread.java:595)

       

      Configuration:

       

      xa-ds.xml

      <xa-datasource>
          <jndi-name>XADDS</jndi-name>
          <track-connection-by-tx>true</track-connection-by-tx>
          <isSameRM-override-value>false</isSameRM-override-value>
          <connection-property name="SetBigStringTryClob">true</connection-property>
          <xa-datasource-class>com.perotsystems.diamond.tools.toplink.OracleXADataSourceFix</xa-datasource-class>
          <xa-datasource-property name="URL">jdbc:oracle:thin:@t710:1521:kqa</xa-datasource-property>
          <xa-datasource-property name="User">xxxxx</xa-datasource-property>
          <xa-datasource-property name="Password">xxxxxxxxx</xa-datasource-property>
          <min-pool-size>20</min-pool-size>
          <max-pool-size>50</max-pool-size>
          <blocking-timeout-millis>5000</blocking-timeout-millis>
          <idle-timeout-minutes>300</idle-timeout-minutes>
          <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
          <container-managed-security/>
          <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
          <check-valid-connection-sql>select 1 from dual</check-valid-connection-sql>
        </xa-datasource>

       

      Expected result:

           All 1000 messages should be successfully processed.

       

      Options already tried to resolve:

       

      1. Increased the java pool size in Oracle, loaded the JVM and the XA libraries in Oracle and this did not help.
      2. Setting the pad attribute to true in conf/jboss-service.xml.  Got the different error
          [java] 20:02:11,944 ERROR [StdServerSession] failed to commit/rollback
          [java] org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=t710/324, BranchQual=, localId=324] status=STATUS_NO_TRANSACTION; - nested throwable: (oracle.jdbc.xa.OracleXAException)
      3. Checked with oracle support.

       

      Sorry for long description, any help would be appreciated.

       

      Thanks in advance,
      Bala

        • 1. Re: oracle.jdbc.xa.OracleXAException during jboss clustering
          Bala Tubati Newbie

          It seems the Oracle XA exception was resolved. 

           

          1) First I try to print the XA exception details by configuring “OracleXAExceptionFormatter” in xadiamond-ds.xml.

           

            <mbean code="org.jboss.resource.adapter.jdbc.vendor.OracleXAExceptionFormatter"

                   name="jboss.jca:service=OracleXAExceptionFormatter">

              <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>

            </mbean>

          The actual exception printed as follows

               [java] 18:28:37,692 WARN  [TransactionImpl] xa error: -3 (A resource manager error has occured in the transaction branch.); oracle error: 24757; oracle sql error: 0;

               [java] oracle.jdbc.xa.OracleXAException

           

          2) Then the link (http://community.jboss.org/wiki/IHaveProblemsWithOracleXA/diff?secondVersionNumber=10) helped a lot.  Updated the xadiamond-ds.xml with the mentioned tags.

           

           

              <track-connection-by-tx/>

              <isSameRM-override-value>false</isSameRM-override-value>

              <no-tx-separate-pools/>

           

          3)  And enabled Pad attribute of XID in jboss-service.xml

           

             <mbean code="org.jboss.tm.XidFactory"

                name="jboss:service=XidFactory">

                <attribute>true</attribute>

             </mbean>

           

          Then it started working as expected.

           

          Thanks and regards,

          Bala

          • 2. Re: oracle.jdbc.xa.OracleXAException during jboss clustering
            Bala Tubati Newbie

            For a complex transaction again observed the same error.  As a work around updated XIDFactory Pad=true for node 1 and Pad=false for node 2, then the generated XID is not going to be same for both the nodes.  This fix worked with complex transactions too.

             

            But the limitation is we can configure only two nodes.  More than two nodes this error may occur again.

             

            Please provide comments on these settings?

             

            Thanks and regards,

            Bala

            • 3. Re: oracle.jdbc.xa.OracleXAException during jboss clustering
              Wolf-Dieter Fink Master

              We use also complex transaction scenario (two oracle DB direct, 3 JCA XA connections).

              One of the problems was that a EAP system use oracle as well, here we have to set 'oracle.jdbc.XATransLoose=true'

               

              I attach my DS configuration, maybe it helps.