oracle.jdbc.xa.OracleXAException during jboss clustering
Hi,
Our application is working as expected with single instance. After configuring the clustering we are observing the Oracle XA error message.
Software Used:
Java Version: 1.5.22(Java HotSpot(TM) 64-Bit Server VM )
JBoss Version: 4.0.2
Database Product Version : Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
JDBC Driver : Oracle JDBC driver 10.2.0.4.0
MQ Server: 6.0.2
Apache mod jk load balancer: httpd-2.2.16
Steps:
1. Shutdown both the nodes.
2. Make sure no messages exist in the queue.
3. Start the node 1 and make sure the node is started.
4. Start the node 2.
5. Run the test program to post 1000 message in queue.
6. Observe the server log of the both the nodes.
Actual result:
Both the nodes are showing Oracle XA Error. . This behavior is always reproducible.
16:05:51,250 WARN [TransactionImpl] XAException: tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=t710/396, BranchQual=, localId=396] errorCode=XAER_RMERR
[java] oracle.jdbc.xa.OracleXAException
[java] at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1120)
[java] at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:249)
[java] at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:131)
[java] at org.jboss.tm.TransactionImpl$Resource.startResource(TransactionImpl.java:1952)
[java] at org.jboss.tm.TransactionImpl.enlistResource(TransactionImpl.java:548)
[java] at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:436)
[java] at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322)
[java] at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:583)
[java] at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450)
[java] at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:874)
[java] at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
[java] at oracle.toplink.jndi.JNDIConnector.connect(JNDIConnector.java:110)
[java] at oracle.toplink.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:147)
[java] at oracle.toplink.internal.databaseaccess.DatasourceAccessor.connect(DatasourceAccessor.java:197)
[java] at oracle.toplink.internal.databaseaccess.DatabaseAccessor.connect(DatabaseAccessor.java:220)
[java] at oracle.toplink.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:400)
[java] at oracle.toplink.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1223)
[java] at oracle.toplink.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:184)
[java] at oracle.toplink.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:485)
[java] at oracle.toplink.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:441)
[java] at oracle.toplink.threetier.ServerSession.executeCall(ServerSession.java:457)
[java] at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:117)
[java] at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:103)
[java] at oracle.toplink.internal.queryframework.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:501)
[java] at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:862)
[java] at oracle.toplink.internal.queryframework.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:837)
[java] at oracle.toplink.queryframework.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:415)
[java] at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:811)
[java] at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:620)
[java] at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:779)
[java] at oracle.toplink.queryframework.ReadObjectQuery.execute(ReadObjectQuery.java:388)
[java] at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2073)
[java] at oracle.toplink.publicinterface.Session.executeQuery(Session.java:988)
[java] at oracle.toplink.publicinterface.Session.executeQuery(Session.java:945)
[java] at oracle.toplink.publicinterface.Session.readObject(Session.java:2588)
[java] at com.perotsystems.diamond.tools.toplink.ToplinkPersistenceWrapper.load(ToplinkPersistenceWrapper.java:629)
[java] at com.perotsystems.diamond.tools.toplink.ToplinkPersistenceWrapper.load(ToplinkPersistenceWrapper.java:597)
[java] at com.perotsystems.diamond.persistence.service.acl.adapter.diamond950.toplink.Diamond950AclPersistenceAdapterImpl.readAcl(Diamond950AclPersistenceAdapterImpl.java:49)
[java] at com.perotsystems.diamond.persistence.service.acl.AclPersistenceServiceImpl.readObject(AclPersistenceServiceImpl.java:101)
[java] at com.perotsystems.diamond.persistence.PersistenceServiceImpl.readObject(PersistenceServiceImpl.java:356)
[java] at com.perotsystems.diamond.persistence.PersistenceServiceImpl.readObject(PersistenceServiceImpl.java:379)
[java] at com.perotsystems.diamond.security.SecurityServiceImpl.loadAcl(SecurityServiceImpl.java:867)
[java] at com.perotsystems.diamond.security.SecurityServiceImpl.login(SecurityServiceImpl.java:324)
[java] at com.perotsystems.diamond.security.SecurityServiceImpl.login(SecurityServiceImpl.java:226)
[java] at com.perotsystems.diamond.security.SecurityServiceSessionBean.login(SecurityServiceSessionBean.java:141)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[java] at java.lang.reflect.Method.invoke(Method.java:592)
[java] at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
[java] at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)
[java] at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
[java] at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:130)
[java] at org.jboss.webservice.server.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:51)
[java] at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
[java] at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
[java] at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
[java] at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
[java] at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
[java] at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
[java] at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
[java] at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:624)
[java] at org.jboss.ejb.Container.invoke(Container.java:873)
[java] at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:413)
[java] at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:88)
[java] at $Proxy786.login(Unknown Source)
[java] at com.perotsystems.diamond.security.clientproxy.SecurityServiceClientProxyEJBLocalImpl.login(SecurityServiceClientProxyEJBLocalImpl.java:106)
[java] at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:89)
[java] at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:112)
[java] at com.perotsystems.diamond.clientproxy.DiamondClientFactoryFacade.login(DiamondClientFactoryFacade.java:101)
[java] at com.perotsystems.diamond.DiamondWebService.login(DiamondWebService.java:275)
[java] at com.perotsystems.diamond.DiamondWebService.getSOAPSecureSession(DiamondWebService.java:222)
[java] at com.perotsystems.diamond.DiamondWebService.getSecureSession(DiamondWebService.java:316)
[java] at com.perotsystems.diamond.ddsservice.ws.DdsServiceSoapBindingImpl.processBatchDirect(DdsServiceSoapBindingImpl.java:592)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[java] at java.lang.reflect.Method.invoke(Method.java:592)
[java] at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
[java] at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
[java] at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
[java] at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
[java] at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
[java] at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
[java] at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
[java] at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
[java] at com.perotsystems.diamond.framework.mdb.ws.WebServiceUtil.invoke(WebServiceUtil.java:107)
[java] at com.perotsystems.diamond.framework.mdb.ws.DispatchProcessor.invokeWebService(DispatchProcessor.java:64)
[java] at com.perotsystems.diamond.framework.mdb.ws.DispatchProcessor.processMessage(DispatchProcessor.java:53)
[java] at com.perotsystems.diamond.framework.mdb.basic.BasicMessageController.processMessage(BasicMessageController.java:66)
[java] at com.perotsystems.diamond.framework.mdb.DiamondMessageDrivenBeanImpl.processMessage(DiamondMessageDrivenBeanImpl.java:208)
[java] at com.perotsystems.diamond.framework.mdb.DiamondMessageDrivenBeanImpl.onMessage(DiamondMessageDrivenBeanImpl.java:175)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[java] at java.lang.reflect.Method.invoke(Method.java:592)
[java] at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
[java] at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:475)
[java] at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
[java] at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:87)
[java] at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
[java] at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
[java] at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
[java] at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
[java] at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94)
[java] at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
[java] at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
[java] at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389)
[java] at org.jboss.ejb.Container.invoke(Container.java:873)
[java] at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1090)
[java] at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
[java] at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256)
[java] at com.ibm.mq.jms.MQSession$FacadeMessageListener.onMessage(MQSession.java:147)
[java] at com.ibm.msg.client.jms.internal.JmsSessionImpl.run(JmsSessionImpl.java:2634)
[java] at com.ibm.mq.jms.MQSession.run(MQSession.java:862)
[java] at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:182)
[java] at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
[java] at java.lang.Thread.run(Thread.java:595)
Configuration:
xa-ds.xml
<xa-datasource>
<jndi-name>XADDS</jndi-name>
<track-connection-by-tx>true</track-connection-by-tx>
<isSameRM-override-value>false</isSameRM-override-value>
<connection-property name="SetBigStringTryClob">true</connection-property>
<xa-datasource-class>com.perotsystems.diamond.tools.toplink.OracleXADataSourceFix</xa-datasource-class>
<xa-datasource-property name="URL">jdbc:oracle:thin:@t710:1521:kqa</xa-datasource-property>
<xa-datasource-property name="User">xxxxx</xa-datasource-property>
<xa-datasource-property name="Password">xxxxxxxxx</xa-datasource-property>
<min-pool-size>20</min-pool-size>
<max-pool-size>50</max-pool-size>
<blocking-timeout-millis>5000</blocking-timeout-millis>
<idle-timeout-minutes>300</idle-timeout-minutes>
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<container-managed-security/>
<valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
<check-valid-connection-sql>select 1 from dual</check-valid-connection-sql>
</xa-datasource>
Expected result:
All 1000 messages should be successfully processed.
Options already tried to resolve:
1. Increased the java pool size in Oracle, loaded the JVM and the XA libraries in Oracle and this did not help.
2. Setting the pad attribute to true in conf/jboss-service.xml. Got the different error
[java] 20:02:11,944 ERROR [StdServerSession] failed to commit/rollback
[java] org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=t710/324, BranchQual=, localId=324] status=STATUS_NO_TRANSACTION; - nested throwable: (oracle.jdbc.xa.OracleXAException)
3. Checked with oracle support.
Sorry for long description, any help would be appreciated.
Thanks in advance,
Bala
