Custom XAResource problems

[jubecker]

Hi,

we encountered some problems with our custom XAResource.

1) We register the XAResource directly with the TransactionManager, we do not have a datasource / JDBC driver! The XAResource is serializable and during our test case, where we crash JBoss (7.1.1), it gets serialized fine to disc. During the recovery we then ran into the following classloading issue:

----

16:35:21,096 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016043: Exception on attempting to restore XAResource: java.lang.ClassNotFoundException: com.versant.odbms.RecoveryXAResource from [Module "org.jboss.jts:main" from local module loader @19c0bd6 (roots: /var/folders/3b/kj_qwqzs1xs_x2mnqmlpt0kcdf_5_z/T/jboss/jboss-as-7.1.1.Final/modules,/var/folders/3b/kj_qwqzs1xs_x2mnqmlpt0kcdf_5_z/T/jboss_modules_1346337224483)]

at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)

at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)

at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)

at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)

at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)

at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)

at java.lang.Class.forName0(Native Method) [classes.jar:1.6.0_33]

at java.lang.Class.forName(Class.java:247) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:603) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) [classes.jar:1.6.0_33]

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) [classes.jar:1.6.0_33]

at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.restore_state(XAResourceRecord.java:922) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.arjuna.coordinator.BasicAction.restore_state(BasicAction.java:1105) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.arjuna.coordinator.BasicAction.activate(BasicAction.java:480) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.arjuna.coordinator.BasicAction.activate(BasicAction.java:443) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.<init>(RecoverAtomicAction.java:50) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:149) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:251) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:]

at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.2.Final.jar:]

16:35:21,108 WARN  [com.arjuna.ats.arjuna] (Periodic Recovery) ARJUNA012154: RecoverAtomicAction: transaction 0:ffffc0a80909:1e870229:503f79ca:12 not activated, unable to replay phase 2 commit. Check state has not already been completed.

----

We could not find anything in the documentation regarding serializable XAResources, only the non-serializable case is covered.

How can we fix this?

2) A hackish/temporary solution for problem 1) is to add our module as a dependency to the org.jboss.jts module. If we do this we ran into another problem:

The method "Xid[] recover(int)" on the XAResource is not called. The first method called is "commit(Xid, boolean)". It is called form RecoverAtomicAction:

RecoveryXAResource.commit(Xid, boolean) line: 120  

XAResourceRecord.topLevelCommit() line: 451

RecoverAtomicAction(BasicAction).doCommit(boolean, AbstractRecord) line: 2753  

RecoverAtomicAction(BasicAction).doCommit(RecordList, boolean) line: 2669  

RecoverAtomicAction(BasicAction).phase2Commit(boolean) line: 1804  

RecoverAtomicAction.replayPhase2() line: 71

AtomicActionRecoveryModule.doRecoverTransaction(Uid) line: 152 

AtomicActionRecoveryModule.processTransactionsStatus() line: 251   

AtomicActionRecoveryModule.periodicWorkSecondPass() line: 109  

PeriodicRecovery.doWorkInternal() line: 789

PeriodicRecovery.run() line: 371   

Digging into the JBoss code I could not find any call to "recover(int)" in RecoverAtomicAction. Looking in to the other RecoveryModules code, I saw that XARecoveryModule is calling it.

I tried to make it the first module to use but failed. I tried with jbossts-properties.xml :

<?xml version="1.0" encoding="UTF-8"?>

<transaction-service>

    <properties depends="arjuna" name="jta">

          <property name="com.arjuna.ats.arjuna.recovery.recoveryExtension1" value="com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule"/>

    </properties>

</transaction-service>

Even patching the default-jbossts-properties.xml did not work. I'm a little bit lost here. :-)

How do I make JBoss use the XARecoveryModule instead of RecoverAtomicAction?

Thanks in advance for any help.

Juergen

[jhalliday]

1) https://issues.jboss.org/browse/JBTM-996 , but you're better off simply not serializing resources, see below.

2) tx resolution paths differ for the top down and bottom up cases. Where a tx log with XAResource exists, commit is called directly on the RecoverAtomicAction path.  A recover scan using XARecoveryModule is needed only to identify tx which are not known to the log, or to rewire logs to XAResources where the log exists but does not contain a serialized resource.  Note that since you need to provide a recovery plugin anyhow for the such cases, you may as well not bother serializing resources into the log. The performance should be better (java object serialization sucks) and you'll see a recover call in all cases rather than just the bottom up ones.

[jubecker]

Thanks a lot, for the pointers and hints!

[jubecker]

We had a look at the link you provided. In the end it boils down to implementing JBoss specific interfaces to make the recovery work.

Are there no other ways to make this work? We would reluctantly introduce a JBoss dependency to our code.

[jhalliday]

The JTA spec does not cover wiring of recovery. We've discussed adding a new api for it in JTA 1.2, but it's more likely to be a later spec release. Until then you're stuck with vendor specific extensions.  The only alternative is a JCA connector, for which IronJacamar will take care of the wiring. That's roughly how 3rd party JDBC and JMS XA drivers get hooked into the AS without needing to supply their own JBoss specific extensions.

[jubecker]

We tried to go the not serializable way and implemented the XAResourceRecovery interface. Searching the documentation we could not find a complete description where to put the property ("com.arjuna.ats.jta.recovery.XAResourceRecoveryTest") to make the implementation known. Some places mention jbossjta-properties.xml, some say jbossts-properties.xml, ...

How does the classloading work for the XAResourceRecovery implementation? How does JBoss figure out which module provides the class?