XA Datasource Recovery in JBoss 5.1.0.

[amodrm]

We use JBoss 5.1.0 with Oracle. Our typical use case has distributed Transaction wherein we update database along with sending JMS messages.
Sometime when machine hosting JBoss 5.1.0 crashes or JBoss server is shutdown abruptly, we are seeing following exception:

"java.sql.SQLException: ORA-01591: lock held by in-doubt distributed transaction .....Error Code: 1591"

Even after restating JBoss, lock held on specific table is not released which might suggest that XA recovery was not initiated by transaction manager.

To resolve this issue, database administrator are manually overriding In-Doubt Transactions with force rollback/commit.

I referred jboss-5.1.0.GA\server\all\conf\jbossts-properties.xml where AtomicActionRecoveryModule,TORecoveryModule,arjunacore.XARecoveryModule are set.
Our assumption was JBoss when restarted will initiate auto recovery of  XA resource.
There are many documents releated to that, but somehow it was not fully clear as whether specific configuration is required in JBoss 5.1.0 for database XA resource?

So my question is do we need to add any configuration for XA Datasource in Jboss 5.1.0 so that Transaction Manager can have auto recovery of XA datasources?

Appreciate any response.

Thanks.

[tomjenkinson]

Hi Amod,

The version of JBoss you are using is older than the ones I am more familiar with. As you suggest, if you use AS7 and configure an xa-datasource recovery should just happen?

Please can you try this on AS7?

Assuming that is not possible, you can of course configure AS5 to do this, it just requires additional recovery configuration (done by default for you in AS7). You could try taking a look at: https://community.jboss.org/wiki/JBossTSRecoveryInAS but this applies to a slightly earlier version of TS. This is also useful: http://management-platform.blogspot.co.uk/2008/11/transaction-recovery-in-jbossas.html. The bits you need to pay particular reference to are: AppServerJDBCXARecovery. You can find some information here
https://issues.jboss.org/browse/JBTM-319 and here http://anonsvn.jboss.org/repos/labs/labs/jbosstm/tags/JBOSSTS_4_2_3_SP5_CP01/atsintegration/classes/com/arjuna/ats/internal/jbossatx/jta/AppServerJDBCXARecovery.java

I can assure you that things have moved on quite a bit since AS5, if you are able to try this on AS7 things should go a lot smoother.

Tom

[amodrm]

Hi Tom,

Appreciate quick response. I have tried all suggested ways in these links. But I am still I getting "lock held by in-doubt distributed transaction .....Error Code: 1591" when I restart the JBoss server.

Things I did: specifically update jbossts-properties.xml

1. Update section <properties depends="arjuna" name="jta"> :

    <property name="com.arjuna.ats.jta.jtaTMImplementation" value="com.arjuna.ats.internal.jta.transaction.jts.TransactionManagerImple"/>         
    <property name="com.arjuna.ats.jta.jtaUTImplementation" value="com.arjuna.ats.internal.jta.transaction.jts.UserTransactionImple"/>

2.  Update section <properties depends="arjuna,txoj,jta" name="recoverymanager">

     <property  name="com.arjuna.ats.arjuna.recovery.recoveryExtension5" value="com.arjuna.ats.internal.jta.recovery.jts.XARecoveryModule"/>

3. I referred jbossjts-integration.jar. It already contains AppServerJDBCXARecovery. So I also tried updating section <properties depends="arjuna,txoj,jta"> :

    <property name="com.arjuna.ats.jta.recovery.XAResourceRecovery1" value= "com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;MyXADataSourceName"/>

4. I also gave special privileges to my XA datasource user.

So what I am trying it, just before phase2Commit() gets called, I shutdown the JBoss server explicitly. After  restart the server, I can see JBoss adding row in table "dba_2pc_pending" with status as prepared. I assume after recovery, this row should get updated. When I acess my application table thereafter through my application, I get error "lock held by in-doubt distributed transaction ..."

Other warnings that I could see in the log are:

a. com.arjuna.ats.internal.jta.resources.arjunacore.noresource] No XAResource to recover < 1...

b. No XA recovery nodes specified. Will only recover saved states.

c. If I add <property name="com.arjuna.ats.jta.recovery.XAResourceRecovery.JBMESSAGING1" value="org.jboss.jms.server.recovery.MessagingXAResourceRecovery;java:/DefaultJMSProvider"/> in section  "<properties depends="arjuna,txoj,jta"

    so that JMS recovery should happen. After I add it & restart the server see following warnings.

com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 26, 24, 494597485249971009758995799975853485755975757985857999748524997100975899579997585348575597575798589748

I referred links for these errors also, but couldn't fix the error .

1. Is there some more steps required in addition to 4 steps mentioned above so that XA recovery happens successfully in JBoss 5.1.0. 

   Most of these links refer JBoss 4.x version. Unfortunately we can't use JBoss 7 for current seup. We need to use JBoss 5.1.0.

2. Does warnings as mentioned in a,b,c suggests any missing settings.

Thanks for any feedback/suggesions.

[tomjenkinson]

I think the problem is:

No XA recovery nodes specified. Will only recover saved states.

As I say, this version predates my expertise with transactions integration with the app server, but from the the actual JBTM code wants to see is:

a value for com.arjuna.ats.jta.xaRecoveryNode with a value set to the node name you have configured for the transaction manager (i.e. via com.arjuna.ats.arjuna.xa.nodeIdentifier).

If you take a look in jbossts-properties.xml, have you set: com.arjuna.ats.jta.xaRecoveryNode

Tom

[amodrm]

Hi Tom,

Yes, I have set these properties as:

<property name="com.arjuna.ats.arjuna.xa.nodeIdentifier" value="1"/>
<property name="com.arjuna.ats.jta.xaRecoveryNode" value="1"/>

I am testing on single server as of now to reproduce this issue.

Thanks,

[tomjenkinson]

Hi Amod,

When you set then there should be no way that the following would be printed:

No XA recovery nodes specified. Will only recover saved states.

(That I can see, that is)

Can you provide the output containing this text in context?

Thanks,
Tom

[amodrm]

Hi Tom,

You are right.  That was old log. But I still get following 2 errors: Does it suggests any other missing configuration?

2012-11-06 16:50:30,246 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] (Thread-26) [com.arjuna.ats.internal.jta.recovery.info.firstpass] JTS XARecoveryModule - first pass
2012-11-06 16:50:40,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-26) Periodic recovery - second pass <Tue, 6 Nov 2012 16:50:40>
2012-11-06 16:50:40,248 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-26) Periodic recovery - second pass <Tue, 6 Nov 2012 16:50:40>
2012-11-06 16:50:40,249 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-26) AtomicActionRecoveryModule: Second pass
2012-11-06 16:50:40,249 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-26) AtomicActionRecoveryModule: Second pass
2012-11-06 16:50:40,261 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (Thread-26) [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 26, 24, 49459748524997100975810154485458534857561005357555855100974852499710097581015448545853485756100535755585649 >
....
2012-11-06 16:50:40,264 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (Thread-26) [com.arjuna.ats.internal.jta.resources.arjunacore.noresource] [com.arjuna.ats.internal.jta.resources.arjunacore.noresource] No XAResource to recover < 131075, 26, 24, 49459748524997100975810154485458534857561005357555855100974852499710097581015448545853485756100535755585649 >

Thanks.

[tomjenkinson]

Hi Amod,

That definitely looks like you haven't got recovery configured quite right. Please can you attach your jbossts-properties.xml file and I will take a look.

Tom

[amodrm]

Hi Tom,

Please find attached jbossts-properties.xm.

Thanks.

[tomjenkinson]

Thanks, please can you confirm the command you are using to start the server?

[amodrm]

Tom,

Here is the command that I use:

jboss-5.1.0.GA\bin>run.bat -c all

[tomjenkinson]

Hi Amod,

I diffed your config against AS 5.1.0 and see that you have:

<property name="com.arjuna.ats.arjuna.xa.nodeIdentifier" value="1"/>

Which is great, but you have it in the section called:

<properties depends="arjuna,txoj,jta" name="recoverymanager">

When it should be in the section:

<properties depends="common" name="arjuna">

Take a look at the stock jbossts-properties.xml and you will see where it was originally.

Hope that resolves it for you,

Tom

[amodrm]

Hi Tom,

Thanks a lot for pointing it out. Even after correcting that, I am still getting the those errors 

I diffed against original file & here are the changes :

1. In section: <properties depends="arjuna" name="jta">
   Updated new values for "com.arjuna.ats.jta.jtaTMImplementation" & "com.arjuna.ats.jta.jtaUTImplementation" as suggested in https://community.jboss.org/wiki/JBossTSRecoveryInAS

2. In section <properties depends="arjuna,txoj,jta",

     Added following new properties

    <property name="com.arjuna.ats.jta.xaRecoveryNode" value="1"/>  

    <property name="com.arjuna.ats.jta.recovery.XAResourceRecovery1" value= "com.arjuna.ats.internal.jbossatx.jta.AppServerJDBCXARecovery;XAOracleDS"/> 

    <property  name="com.arjuna.ats.arjuna.recovery.recoveryExtension5" value="com.arjuna.ats.internal.jta.recovery.jts.XARecoveryModule"/>

I have attached updated file.

Errors that I get are same i.e.

a. No XAResource to recover ....&

b. Could not find new XAResource to use for recovering non-serializable XAResource ...

Thanks.

[tomjenkinson]

Hi Amod,

It appears you are using JTS. I assume that is because you have multiple application server nodes, correct?

Tom

[tomjenkinson]

Also, as well as confirming you are intending to use JTS, please can you confirm you are using the jbossts-properties.xml in ./server/all/conf/jbossts-properties.xml