disable XA

[mazz]

Within my JBossAS 4.2.1 app, I'm getting the infamous:

[com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource

I'm not sure what two data sources are being enlisted in this transaction to make it want to be XA-recovered. The only two data sources I have deployed are a quartz <no-tx-datasource> data source and my main app's <local-tx-datasource> data source. I assume something is going on within a Quartz job causing this but no where in my Quartz config do I mention the use of XA or a XA datasource. I also have JMS beans that sometimes hiccup, and I believe the JMS subsystem out of box IS configured to somehow use XA - I don't know if that is somehow involved. The only place where I see XA mentioned is in the JMS configuration.

In any event, I don't want anything in my app to go through XA or be in a 2PC transaction. Nothing can be recoverable - in fact, if I shutdown my server, delete the entire data/tx-object-store directory within my JBossAS and restart, everything works fine (database is not in a bad state - I may not have data committed that tried to be commited before, but my app can work fine - the business layer can survive this). Therefore, I don't want the Tx manager to even attempt to recover, since (a) it can never recover and (b) I'm fine regardless of the tx error.

I scoured the 'net but can't find any good docs that explicitly talk about how you can configure JBossAS and its tx manager to NOT attempt to perform this XA recovery. Or, how I can remove the XA component out of the JMS configuration.

Does anyone know how to do any of this? Or know of some docs that relate to this? I looked through http://www.jboss.org/community/docs/DOC-12463 but didn't see anything obvious... but then, I barely know how to spell XA, so I might have missed something.

[marklittle]

Sorry, but we don't encourage you to cripple the transaction engine. It's an ACID transaction manager and works really hard to give you the guarantees that means. I assume you realise that if you are seeing this message it means you had a crash (or a reboot) during the commit phase of the transaction? That's usually not a good thing to ignore, so I'd check what is happening there first to ensure that you understand fully before going down the route of taking recovery out of the picture.

Now if you are really really really sure that you don't want recovery you probably need to ask yourself why you're using transactions at all. If you get past that question and are really really really really sure that you somehow need the overhead of coordination (yes, there's overhead involved even if you manage to register multiple one-phase resources in the same transaction), then you can always look to remove the XARecovery module from the jbossts property file. But if you go down that route don't come and ask for help in driving recovery manually when/if you need to ;-)

[marklittle]

BTW, if you do disable XARecovery then you will find that your ObjectStore builds up logs that cannot be recovered. If that happens then I would recommend that you figure out why you think you don't need transactions when the system you are using clearly does.

[mazz]

Well, here's the general point with this and why I'm asking. Yes, I have seen we are having problems committing data - I still haven't figured out why this is happening (not repeatable - only happens under high load in large environments - still trying to replicate).

But, when this does happen (and even if I had prestine code that worked fine all the time, this can still happen if someone trips over the machines's power cord during the commit phase), we can't recover anyway - that's what this Arjuna error message is saying I guess. So, my question is - why bother in the first place then? If I know that the transaction manager is going to tell me its recovery is going to fail ALL the time (there is absolutely no chance it can ever succeed if I read that error message correctly - our XAResource is not serializable and not ever recoverable), then why should I have it try? And when it does, the ObjectStore still seems to build up over time (does this recovery manager ever stop attempting to recover a particular tx? or does it keep trying infinitum to recover these unrecoverable transactions?

That's what I'm trying to solve - if I can't ever recover anyway, then I don't want it to try.

[marklittle]

"mazz" wrote:

But, when this does happen (and even if I had prestine code that worked fine all the time, this can still happen if someone trips over the machines's power cord during the commit phase), we can't recover anyway - that's what this Arjuna error message is saying I guess.

Not quite. What the message is saying is that we're trying to recover something and we can't because we don't have the right plugin for that resource. So you (as the deployer) need to resolve that. That resolution could require you to write the plugin yourself or work with the resource manager provider to get them to do it.

So, my question is - why bother in the first place then?

Assuming this is not an existential question, the answer is that you should bother to find the right resolution. If the airbag on your car stop working you'd get it fixed pretty quick, right?

If I know that the transaction manager is going to tell me its recovery is going to fail ALL the time (there is absolutely no chance it can ever succeed if I read that error message correctly - our XAResource is not serializable and not ever recoverable), then why should I have it try?

No, your reading the message incorrectly. The wiki (http://www.jboss.org/community/docs/DOC-12463) is pretty clear as to what this message means and it definitely doesn't mean what you think it does.

And when it does, the ObjectStore still seems to build up over time (does this recovery manager ever stop attempting to recover a particular tx?

It will eventually move the log elsewhere and require manual intervention.

or does it keep trying infinitum to recover these unrecoverable transactions?

That's what I'm trying to solve - if I can't ever recover anyway, then I don't want it to try.

Hopefully you'll understand why your assumption about this is wrong and why you need to fix the recovery problem.

[mazz]

So you (as the deployer) need to resolve that. That resolution could require you to write the plugin yourself or work with the resource manager provider to get them to do it.

So, what I think you are saying when you say "work with the resource manager provider" is that our JDBC driver must support this functionality and if not, we need to get one that does... correct? :}

I assume the reason why this happens is that my JDBC driver does not provide the appropriate implementation of XAResource?

The JDBC driver that is in use when this occurs is the Oracle Thin Driver v10.2.0.4.0: http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_10201.html

Let's use a concrete example here - if I am using Oracle 10g, with that JDBC driver, and my app is a J2EE/EJB3 app running in JBossAS 4.2.1 - what specifically would I need to do to get this recovery to happen successfully?

[marklittle]

"mazz" wrote:

So you (as the deployer) need to resolve that. That resolution could require you to write the plugin yourself or work with the resource manager provider to get them to do it.

So, what I think you are saying when you say "work with the resource manager provider" is that our JDBC driver must support this functionality and if not, we need to get one that does... correct? :}

Well it needs to use an XAResource that is supported by the TM for recovery purposes.

I assume the reason why this happens is that my JDBC driver does not provide the appropriate implementation of XAResource?

It's either not Serializable or it's not supported by one of the XAResourceRecovery implementations we ship.

The JDBC driver that is in use when this occurs is the Oracle Thin Driver v10.2.0.4.0: http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_10201.html

Let's use a concrete example here - if I am using Oracle 10g, with that JDBC driver, and my app is a J2EE/EJB3 app running in JBossAS 4.2.1 - what specifically would I need to do to get this recovery to happen successfully?

Make sure that the Oracle XAResourceRecovery implementation is in place.

[mazz]

OK, the fog is beginning to clear.

http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Transactions_JTA_Programmers_Guide/index.html#sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Shipped_XAResourceRecovery_implementations

The above mentions this Oracle and "generic JNDI" recovery modules. This shows the properties to set within the [transaction-service> tag, which is done inside the conf/jbossjta-properties.xml.

Now, based on that, and looking at the out-of-box jbossjta-properties.xml that came with JBossAS 4.2, I will infer that in order to provide an Oracle resource recovery object, I need to add this to my jbossjta-properties.xml (note: out of box, the [properties name="jdbc"> is already in here, though it doesn't have the type="system" attribute but it does have 'depends="jta"' - I will leave it as-is and just add the child [property> elements):

<properties depends="jta" name="jdbc">
 <!--
 property name="com.arjuna.ats.jdbc.isolationLevel" value="TRANSACTION_SERIALIZABLE"/>
 -->

 <!-- <property name="DatabaseURL" value="jdbc:oracle:thin:@my.hostname:1521:mySID"/> -->
 <property name="DatabaseJNDIName" value="java:/MyDS"/>
 <property name="UserName" value="my-db-username"/>
 <property name="Password" value="my-db-password"/>
 </properties>

Note that my app will support either PostgreSQL or Oracle, so I will not use the DatabaseURL property, but instead, rely on the DatabaseJNDIName property and let the -ds.xml configuration define the type of DB the app connects to.

I then have to tell the TM to put the recovery module in place. So, in the existing list of recovery objects, I add the new one:

<!--
 Periodic recovery modules to use. Invoked in sort-order of names.
 -->
 <property
 name="com.arjuna.ats.arjuna.recovery.recoveryExtension1" value="com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule"/>
 <property
 name="com.arjuna.ats.arjuna.recovery.recoveryExtension2" value="com.arjuna.ats.internal.txoj.recovery.TORecoveryModule"/>
 <property
 name="com.arjuna.ats.arjuna.recovery.recoveryExtension3" value="com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule"/>
 <!-- ADD THIS TO PROVIDE RECOVERY FOR JNDI ACCESSIBLE DATA SOURCES -->
 <property
 name="com.arjuna.ats.arjuna.recovery.recoveryExtension4" value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery"/>

I question if I even need those first three - can I get rid of those? Obviously, they aren't helping me now :)

I think JBossAS should at least ship with commented out versions of the above in its jbossjta-properties.xml to help people get a head start on this configuration.

As a side note, this page tells you how the TM will look for recovery modules via system properties:

http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp03/html/Transactions_JTA_Programmers_Guide/sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Recovering_XAConnections.html

[mazz]

I tried the above but when I run JBossAS I get this:

16:31:12,500 WARN [arjLoggerI18N] [com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery_2]
- Recovery module com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery
does not conform to RecoveryModule interface

I had this in my jbossjta-properties.xml:

<property name="com.arjuna.ats.arjuna.recovery.recoveryExtension4"
value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery"/>

which is incorrect.

Looking at the code, it looks like this is because of a ClassCastException after instantiating that class and casting to RecoveryModule. So, I am mistaken about how to deploy this.

The docs at http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Transactions_JTA_Programmers_Guide/index.html#sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Shipped_XAResourceRecovery_implementations state that "com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery" is the name of a valid recovery class, but I thought it was saying that was the name of the recovery module.

Looking at the source, this is not a RecoveryModule, it is an XAResourceRecovery object: http://anonsvn.labs.jboss.com/labs/jbosstm/tags/JBOSSTS_4_3_GA/ArjunaJTA/jdbc/classes/com/arjuna/ats/internal/jdbc/recovery/JDBCXARecovery.java

I need to rely on what was here out of box (specifically, I see there is already the "com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule" module deployed).

I think the answer is here: http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Transactions_JTA_Programmers_Guide/index.html#sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Recovering_XAConnections

I need to set a property that defines this resource recovery object:

<property name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC"
value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery"/>

The docs say, "These properties need to go into the JTA section of the property file" which I assume means I put my recovery property in this section:

<properties depends="arjuna" name="jta">

[mazz]

Getting closer:

2008-11-25 17:45:27,593 ERROR [STDERR] java.lang.NullPointerException
2008-11-25 17:45:27,593 ERROR [STDERR] at javax.naming.InitialContext.getURLScheme(InitialContext.java:269)
2008-11-25 17:45:27,593 ERROR [STDERR] at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:318)
2008-11-25 17:45:27,593 ERROR [STDERR] at javax.naming.InitialContext.lookup(InitialContext.java:392)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery.createDataSource(JDBCXARecovery.java:174)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery.hasMoreResources(JDBCXARecovery.java:141)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:679)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:179)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:237)
2008-11-25 17:45:27,593 ERROR [STDERR] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)

Clearly, I'm not defining my properties correctly or in the same spot (it doesn't see my DatabaseJNDIName property). This is why jbossjta-properties.xml needs to have sections commented out in here to show how it should be done. I wrote up a JIRA on this so the JBossAS guys can provide some additional help here: https://jira.jboss.org/jira/browse/JBAS-6244

[mazz]

I've tried several different ways to configure this in my jbossjta-properties.xml to no avail - the JDBCXARecovery class keeps triggering a NullPointerException because its passing null as the JNDI name to the InitialContext. I'm refering to this page in the docs: http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Transactions_JTA_Programmers_Guide/index.html#sect-Transactions_JTA_Programmers_Guide-Transaction_Recovery-Shipped_XAResourceRecovery_implementations

What's the secret to getting my properties to be seen by the XARecovery object?

I tried these different ways:

1) use type="System" (does this put the properties in the System.getProperties() props?)

<property name="DatabaseJNDIName" value="java:/MyDS" type="System"/>
<property name="UserName" value="username" type="System"/>
<property name="Password" value="password" type="System"/>

2) Without any type= attribute, just specifying the props like the others

<property name="DatabaseJNDIName" value="java:/MyDS"/>
<property name="UserName" value="username"/>
<property name="Password" value="password"/>

3) I put the properties in the [properties depends="jta" name="jdbc"> section - and I also tried putting them in the [properties depends="arjuna" name="jta"> section (I think the latter section is where it should go, at least, the docs say that)

[mazz]

This is bugging me so much that I decided to just grab the code and step thru it in a debugger - I'm seeing something that is odd, but it's how it works on my JBossAS 4.2.1 install. It isn't working like it is supposed to work (i.e. how the docs say it should work).

See this code in JDBCXARecovery - notice how it will never get properties from the property manager UNLESS you give a parameter in the recovery property itself (like "...JDBCXARecovery;foo"):

public boolean initialise(String parameter)
throws SQLException {
...
 if (parameter == null)
 return false;
 try {
 jdbcPropertyManager.propertyManager.load(XMLFilePlugin.class.getName(), parameter);
 _props = jdbcPropertyManager.propertyManager.getProperties();
 _dbName = _props.getProperty(DATABASE_JNDI_NAME);
 _user = _props.getProperty(USER_NAME);
 _password = _props.getProperty(PASSWORD);
 }

Therefore, unless a parameter is defined in the property value for the recoverer, this JDBCXARecovery will not know what _dbName, _user and _password to use (you get a NullPointerException when it tries to perform a lookup from the initial context, as what happened to me).

Now, look at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule - you can see where it calls this initialise() method for all its recoverers; this code is found in its constructor:

// see if there is a string parameter
int breakPosition = theClassAndParameter.indexOf(BREAKCHARACTER);
String theClass = null;
String theParameter = null;

if (breakPosition != -1) {
 theClass = theClassAndParameter.substring(0, breakPosition);
 theParameter = theClassAndParameter.substring(breakPosition + 1);
} else {
 theClass = theClassAndParameter;
}
...
 XAResourceRecovery ri = (XAResourceRecovery) c.newInstance();
 if (theParameter != null)
 ri.initialise(theParameter);
 _xaRecoverers.addElement(ri);

Therefore, initialise() is never called unless you provide a ";param" parameter to the recoverer in jbossjta-properties.xml - and if initalise() isn't called, as shown above, JDBCXARecovery can never work!

You have to define the recoverer in this odd way to get it to work:

<property name="com.arjuna.ats.jta.recovery.XAResourceRecoveryJDBC" value="com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery;foo"

You need some value (anything) as the parameter. The value of this parameter is passed as a URI to the property manager - it can be anything - internally XMLFilePlugin.load will fail to load that URI but at least it'll get initialise() to be called. You can set this param literally to "jbossjta-properties.xml" and it'll just reload that properties file again. The entire jbossjta-properties set of properties are then given to the recoverer. This seems like a bug in the code to me.

Anyhoo, I'm slowly but surely jumping over the final hurdles here. Once I did the above, I managed to get this recoverer to be invoked, but it turns out my data source is a [local-tx-datasource> and thus not a XADataSource (which causes ClassCastExceptions in this JDBCXARecovery object).

I'm curious to know how many people have actually configured their JBossAS J2EE apps' JDBC data sources to be XA-recoverable. This has not been intuitive and as shown above, can't actually work as documented without sprinkling alittle "foo" in the config :)

[marklittle]

I question if I even need those first three - can I get rid of those? Obviously, they aren't helping me now :)

You don't need the TO recovery module but you do need the AA one since that's what drives the transaction recovery top-down.

[marklittle]

"mazz" wrote:
I tried the above but when I run JBossAS I get this:

16:31:12,500 WARN [arjLoggerI18N] [com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery_2]
- Recovery module com.arjuna.ats.internal.jdbc.recovery.JDBCXARecovery
does not conform to RecoveryModule interface

That's because it's not a RecoveryModule, but an XAResourceRecovery plugin that is used by the XARecoveryModule (which is in the properties file.)

[marklittle]

Take a look at what the TS docs say about BasicXARecovery. I think the JDBCXARecovery was an update of that and we forgot to document it similarly (BasicXARecovery was really just meant to be an example/template).