Problem configuring Oracle for XA Recovery| JBoss.org Content Archive (Read Only)

15. Re: Problem configuring Oracle for XA Recovery

scarceller Jun 25, 2009 11:15 AM (in response to abonfield)

"jhalliday" wrote:
> Please help, we have tried everything we could think of.

Which includes configuring AppServerJDBCXARecovery, right? Send me the debug logs and I'll take a look. Better still, upgrade to 5.1.0.GA, rerun the test, then send me the logs from that.

No problem I'll try the V5.1GA.

But few questions:

1) Is the V5.1 'out-of-box' setup for doing this? or do I need to configure something special to allow the TM to recover in-doubts from Oracle? You mentioned something about 'AppServerJDBCXARecovery' is this the default setup for 5.1GA?

2) I'm also using the default JTA and not JTS, my transaction scope is single server, meaning I do not call any other J2EE components (EJBs or the like) outside this single server. The app simply has a jsp that calls a EJB3 SSB which then calls Entity EJBs (EJB3 Pojos) that interact with 2 databases, one is DB2 the other is Oracle. Both DBs have XA JDBC connections configured and the issue is when we get an in-doubt by means of one of our test cases designed to produce the in-doubt in Oracle we notice that JBoss can not cleanup the in-doubt. The erros seem to indicate that the Tran Mgr can NOT build the recovery connection needed to the Oracle DB.

I will re-test on V5.1GA and let you know what I find.

16. Re: Problem configuring Oracle for XA Recovery

scarceller Jun 26, 2009 12:06 PM (in response to abonfield)

"jhalliday" wrote:
> Please help, we have tried everything we could think of.

Which includes configuring AppServerJDBCXARecovery, right? Send me the debug logs and I'll take a look. Better still, upgrade to 5.1.0.GA, rerun the test, then send me the logs from that.

OK, I want to configure the AppServerJDBCXARecovery class and the docs I see keep talking about a class named:

com.arjuna.ats.internal.jdbc.recovery.AppServerJDBCXARecovery

The issue is NO such Class with that package name exists in JBossAS V5.1 I ran a search of ALL *.jar files for class "AppServerJDBCXARecovery" and I did find this class but it has a different package name, it is:

com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery

and it's located in the following jar file:

..\common\lib\jbossjts-integration.jar

Is this the correct class?
if not how do I get the jar file with the class: com.arjuna.ats.internal.jdbc.recovery.AppServerJDBCXARecovery?

The docs are very difficult to understand. Please help.

17. Re: Problem configuring Oracle for XA Recovery

jhalliday Jun 26, 2009 2:26 PM (in response to abonfield)

> ...the docs I see ...

where?

18. Re: Problem configuring Oracle for XA Recovery

scarceller Jun 26, 2009 3:59 PM (in response to abonfield)

"jhalliday" wrote:
> ...the docs I see ...

where?

My mistake, you are correct I can not find any docs on howto correctly install and configure the AppServerJDBCXARecovery class, only pieces and parts to the process.

I found these:
https://www.jboss.org/community/wiki/JBossTSRecoveryInAS
https://jira.jboss.org/jira/browse/JBTM-319
https://jira.jboss.org/jira/browse/JBTM-501
and the most useful explanation of all was from this fella
http://management-platform.blogspot.com/2008/11/transaction-recovery-in-jbossas.html

In the end, I could not find clear documentation for howto get the JBossAS TM configured correctly to recover in-doubt transactions against Oracle. This is most surprising, if you are doing XA processing it is simply a mater of time before an XA application will run up against this issue. I think I'm set for now but it took the better part of 2 weeks to figure this out.

Thanks for the help.

19. Re: Problem configuring Oracle for XA Recovery

jhalliday Jun 26, 2009 5:03 PM (in response to abonfield)

ok, so if you want the fully qualified class name fixed you need to talk to the author of the blog. It's not part of the official docs or wiki, so the JBossTS team have no control. As for more official docs, what step of the recovery install process is missing from the AppServerJDBCXARecovery javadoc? That I do control and will be happy to improve. Right now the only problem I see is that it has not been fully updated and still refers to the old jbossjta-properties.xml name rather than the newer jbossts-properties.xml

/**
* This provides recovery for compliant JDBC drivers accessed via datasources deployed in JBossAS 5
* It is not meant to be db driver specific.
*
* This code is based on JDBCXARecovery, which expects JNDI to contain an XADataSource implementation.
* In JBossAS, the object created in JNDI when a -ds.xml file containing <xa-datasource> element is used,
* does not implement the XADataSource interface. We therefore use this modified code to pull the
* datasource configuration information from the app server's JMX and instantiate an XADataSource from
* which it can then create an XAConnection.
*
* To use this class, add an XAResourceRecovery entry in the jta section of jbossjta-properties.xml
* for each database for which you need recovery, ensuring the value ends with ;jndiname=<datasource-name>
* i.e. the same value as is in the -ds.xml jndi-name element.
*
* Note for users with secured JMX invokers, use the extended format:
* ;jndiname=MyExampleDbName,username=foo,password=bar
* The username and password refer to the JMX invoker, NOT the datasource.
*
* It's not possible to override the jdbc username and password given in the -ds.xml file at present.
* Since the recovery system sometimes needs greater db user privs than the app code, it may be preferable to
* set up a -ds.xml file only for recovery usage. This approach works for databases which allow users to
* recover one another's transactions, which is most of them. But consult
* your db docs or friendly neighbourhood DBA to be sure, then test it anyhow.
*
* You also need the XARecoveryModule enabled and appropriate values for nodeIdentifier and xaRecoveryNode set.
* See the JBossTS recovery guide if you are unclear on how the recovery system works.
*
* Note: This implementation expects to run inside the app server JVM. It probably won't work
* if you configure the recovery manager to run as a separate process. (JMX can be accessed remotely,
* but it would need code changes to the lookup function and some classpath additions).
*
*
* <properties depends="arjuna" name="jta">
* ...
* <property name="com.arjuna.ats.jta.recovery.XAResourceRecovery1"
* value= "com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;jndiname=MyExampleDbName[,username=foo,password=bar]"/>
* 
* <property name="com.arjuna.ats.jta.xaRecoveryNode" value="1"/>
*
*/

20. Re: Problem configuring Oracle for XA Recovery

scarceller Jun 26, 2009 5:18 PM (in response to abonfield)

Thanks for the doc header from the source code, yes this is exactly what I needed 2 weeks ago. How does one find this stuff?

I agree that posts on forums from other folks are not the best way to figure this stuff out. I just never could find what you just gave me. I even learned something new from what you just provide, I've been wondering if I could pass userid and pswd for recovery of in-doubts and now I see you can. Very good.

Thanks so much for this most informative post.

21. Re: Problem configuring Oracle for XA Recovery

scarceller Jul 8, 2009 2:12 PM (in response to abonfield)

Johnathan,

I have configured my JBossAS V5.1 with the AppServerJDBCXARecovery class and associated the class with the XA JDBC Datasource for my Oracle 10g database.

The testcase I'm doing intentionally produces an in-doubt transaction in the Oracle DB and I know the JBossAS TM knows that this in-doubt exists because before I configured the AppServerJDBCXARecovery class the TM would poll about ever 90seconds and say it could not build a connection back to the DB. After I configured the AppServerJDBCXARecovery I still see this same polling so long as the network stays disconnected to the DB. However once the Oracle DB comes back on-line the in-doubt remains and the polling ends in the TM. Almost like the TM thinks it resolved the in-doubt but it has not.

Then while turning on debug and looking through the logs I noticed that 2 recovery managers are trying to cleanup/recover transaction:
- They both go into 2nd phase recovery.
- I see XARecoveryModule doing 2nd pass
- I also see TORecoveryModule doing 2nd pass

Keep in mind the application being tested only uses 2 data sources and both are XA so the question is: should TORecoveryModule also be recovering something if all transactions are 2PC XA?

Here is the log after the server starts up:

2009-07-07 17:46:57,482 INFO [org.apache.coyote.http11.Http11Protocol] (main) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2009-07-07 17:46:57,529 INFO [org.apache.coyote.ajp.AjpProtocol] (main) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2009-07-07 17:46:57,545 INFO [org.jboss.bootstrap.microcontainer.ServerImpl] (main) JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 1m:22s:485ms
------ right after startup it tries recovery and only tries this once after this
------ no further recovery is attempted and the in-doubt remains in the db
2009-07-07 17:47:46,842 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - first pass <Tue, 7 Jul 2009 17:47:46>
2009-07-07 17:47:46,842 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2009-07-07 17:47:46,842 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] (Thread-12) [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2009-07-07 17:47:46,842 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) StatusModule: first pass
2009-07-07 17:47:56,904 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - second pass <Tue, 7 Jul 2009 17:47:56>
2009-07-07 17:47:56,904 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2009-07-07 17:47:56,935 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) XA datasource does not support isValid method - connection will always be recreated
2009-07-07 17:47:56,967 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) Created new XAConnection
2009-07-07 17:47:56,982 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.rollingback] Rolling back < 131075, 30, 28, 4945455110253551025755995849515148585297535198101559958569948514551102535510257559958495151485852975351981015599585610150102 >
2009-07-07 17:47:56,982 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] (Thread-12) [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2009-07-07 17:47:56,982 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) AtomicActionRecoveryModule: Second pass
2009-07-07 17:47:56,998 WARN [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 30, 28, 49454551102535510257559958495054495852975351985110098584953985245511025355102575599584950544958529753519851100985849559849 >

22. Re: Problem configuring Oracle for XA Recovery

jhalliday Jul 9, 2009 4:29 AM (in response to abonfield)

"scarceller" wrote:

The testcase I'm doing intentionally produces an in-doubt transaction in the Oracle DB and I know the JBossAS TM knows that this in-doubt exists because before I configured the AppServerJDBCXARecovery class the TM would poll about ever 90seconds and say it could not build a connection back to the DB.

Actually it will poll every two minutes by default. Also, just because JBossTS has an in-doubt tx in its log, that does not mean it's the same one that's in-doubt in Oracle. For a proper test case, start with an empty object store and clean db.

"scarceller" wrote:

After I configured the AppServerJDBCXARecovery I still see this same polling so long as the network stays disconnected to the DB. However once the Oracle DB comes back on-line the in-doubt remains and the polling ends in the TM. Almost like the TM thinks it resolved the in-doubt but it has not.

Does it say its found and resolved the in-doubt branch? At debug level the logs will show how many in-doubt xids it got from the db recover scan and what it did with them.

"scarceller" wrote:

Then while turning on debug and looking through the logs I noticed that 2 recovery managers are trying to cleanup/recover transaction:
- They both go into 2nd phase recovery.
- I see XARecoveryModule doing 2nd pass
- I also see TORecoveryModule doing 2nd pass

That's two modules, not two recovery managers.

"scarceller" wrote:

Keep in mind the application being tested only uses 2 data sources and both are XA

So where is the recovery config for the other one? You only mentioned configuring one.

"scarceller" wrote:

so the question is: should TORecoveryModule also be recovering something if all transactions are 2PC XA?

Only if you use transactional objects.

Jonathan.

23. Re: Problem configuring Oracle for XA Recovery

scarceller Jul 9, 2009 10:06 AM (in response to abonfield)

Jonathan,

Before we run the test we insure both DBs have no in-doubts. Then we apply load and simply start plugging and un-plugging the network on Oracle DB till we get the in-doubt. We then stop the load generator. We leave the DB network down and we see the polling, you are correct about 2minute interval, I have changed the polling interval in the config to 1 minute for testing and this seems to cause polling at about 90 seconds or so.

While polling I can see that the tran ids match, the tran id in the log is the same as the in-doubt in Oracle dba_pending_transactions table.

Then what I do is stop the app server clear the app server log files (but not the tran directory (Object Store). Next I re-connect the network to Oracle and finally restart the AppServer, right after the server starts I see the TM tries to recover the tran from Oracle but it only does this once and then never tries again and the in-doubt remains in Oracle.

As for that other extra recovery attempt In the log I posted I think you are correct, it looks like it was a rollback attempt to the other DB which I did not configure the recovery connection, that makes sense.

Also, it's good to know that you mention if the in-doubt recovery really happens something gets logged, who logs this? AppServerJDBCXARecovery class or the JTA layer?

Also, do you know of any good docs on howto view interpret the recovery tran logs in the directory? Or better yet is there some GUI available to see what transactions the AppServer is left dealing with for recovery?

Thanks for your help.

24. Re: Problem configuring Oracle for XA Recovery

jhalliday Jul 9, 2009 10:26 AM (in response to abonfield)

> right after the server starts I see the TM tries to recover the tran from Oracle but it only does this once and then never tries again and the in-doubt remains in Oracle.

Post the logs from that please.

25. Re: Problem configuring Oracle for XA Recovery

mazz Jul 14, 2009 8:52 AM (in response to abonfield)

Hey guys - I'm the author of that blog mentioned earlier.

The FQ class name of that AppServerJDBCResourceRecovery changed and so did the configuration file? Are those changes only relevent to the JBossAS 5 server or did something change in the JBossAS 4.2 line?

FWIW, that blog was written and targeted towards JBossAS 4.2.1 users - I don't know what changed thereafter. I am now on JBossAS 4.2.3 and am wondering if I have to worry about name changes and config file renames.

26. Re: Problem configuring Oracle for XA Recovery

scarceller Jul 22, 2009 11:24 AM (in response to abonfield)

Jonathan,

I just got back to this issue.

I know I have the AppServerJDBCXARecovery class now correctly configured and it creates the connection to the DB just fine now. However, the testcase I'm doing commits the tran to the 1st DB and then I cause a intentional network failure on the 2nd DB before the commit signal is sent to the 2nd DB. The end result is the work is committed to the 1st DB and the tran in the 2nd DB is in-doubt. The 2nd DB is Oracle and I see the in-doubt in the 'dba_2pc_pending' view in Oracle. Under this condition the correct action for the TM is to 'commit' the in-doubt during log processing but this does not occur. I turned on 'DEBUG' tracing on "com.arjuna" and I can see in the log that the TM keeps trying to recover this in-doubt but it seems to say it's trying to do a 'rollback' rather than a 'commit'. I also verified that the tran # in the log matches the tran in the DB and they match, so I'm certain it's the correct tran it's trying to resolve. I'm not sure why the TM thinks a 'rollback' should be done? Even though it's trying a rollback it has no effect and the tran remains in-doubt in the DB?

I even stop the JBossAS server and re-start it and the TM still keeps trying to recover the tran but can't. Also, I have given the correct grants and execute permissions to the needed views in Oracle, I know I have these set OK because other apps are able to recover in-doubts using same userid/password against this same exact DB.

Here are the lines from the log that simply repeat over and over even after the DB communications is re-established:

2009-07-22 11:18:33,312 INFO [org.jboss.bootstrap.microcontainer.ServerImpl] (main) JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221053)] Started in 1m:10s:750ms

2009-07-22 11:18:43,468 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - first pass <Wed, 22 Jul 2009 11:18:43>
2009-07-22 11:18:43,468 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2009-07-22 11:18:53,484 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - second pass <Wed, 22 Jul 2009 11:18:53>
2009-07-22 11:18:53,484 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2009-07-22 11:18:53,499 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) XA datasource does not support isValid method - connection will always be recreated
2009-07-22 11:18:53,531 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) Created new XAConnection
2009-07-22 11:18:53,531 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.rollingback] Rolling back < 131075, 28, 26, 49454551102535557575349585454995852975455495752100585451101455110253555757534958545499585297545549575210058555356 >
2009-07-22 11:18:53,546 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) XA datasource does not support isValid method - connection will always be recreated
2009-07-22 11:18:53,624 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) Created new XAConnection

2009-07-22 11:19:08,671 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - first pass <Wed, 22 Jul 2009 11:19:08>
2009-07-22 11:19:08,671 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2009-07-22 11:19:18,671 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-12) Periodic recovery - second pass <Wed, 22 Jul 2009 11:19:18>
2009-07-22 11:19:18,671 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass
2009-07-22 11:19:18,687 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) Created new XAConnection
2009-07-22 11:19:18,687 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-12) [com.arjuna.ats.internal.jta.recovery.info.rollingback] Rolling back < 131075, 28, 26, 49454551102535557575349585454995852975455495752100585451101455110253555757534958545499585297545549575210058555356 >
2009-07-22 11:19:18,749 DEBUG [com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery] (Thread-12) Created new XAConnection

27. Re: Problem configuring Oracle for XA Recovery

jhalliday Jul 22, 2009 11:32 AM (in response to abonfield)

Turn debug level logging on and send me the full output.

28. Re: Problem configuring Oracle for XA Recovery

marklittle Jul 22, 2009 11:39 AM (in response to abonfield)

Also try to confirm (if debugging doesn't) that the Xid you want to be committed is returned from the XAResource that the recovery module uses.

29. Re: Problem configuring Oracle for XA Recovery

scarceller Jul 22, 2009 11:40 AM (in response to abonfield)

"jhalliday" wrote:
Turn debug level logging on and send me the full output.

Jonathan,

I have set DEBUG on the "com.arjuna" in the jboss-log4j.xml file do you need more than this? If so what do you want me to set?

Then how do you want me to send this large server.log file? Just post the file here? or email it somewhere?

Please, let me know what you want configured for debug and howto send you the log files.

Thanks for your help.