Non Serialized XAResources Warnings with JBoss 4.2, PostgreS

[jantzen]

I'm trying to migrate from 4.0.5 to 4.2, and I'm hitting problems with my datasources. I learned in this process that 4.2 does not support multiple local TX datasources (http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3988662). So, I changed my datasources to be XA's. But, now I get these warnings every few minutes for all three of my datasources.

21:31:10,147 WARN [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 28, 26, 1-7f000001:ba8a:469ed3e6:2097f000001:ba8a:469ed3e6:213

My questions:
What does this mean?
What problems will this lead to?
Is there an inexpensive solution?

Thanks

[jantzen]

Yay for title truncation. That's PostgreSQL 8.1.9

[marklittle]

"jantzen" wrote:
I'm trying to migrate from 4.0.5 to 4.2, and I'm hitting problems with my datasources. I learned in this process that 4.2 does not support multiple local TX datasources (http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3988662).

That's an old article. It does now.

http://wiki.jboss.org/wiki/Wiki.jsp?page=Multiple1PC

So, I changed my datasources to be XA's. But, now I get these warnings every few minutes for all three of my datasources.

21:31:10,147 WARN [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 28, 26, 1-7f000001:ba8a:469ed3e6:2097f000001:ba8a:469ed3e6:213

My questions:
What does this mean?

It means that you've had a crash and the TM is trying to recover. Unfortunately it has a stale XAResource reference and needs you to provide a new one so it can drive recovery.

Check out

http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossTSRecovery

and the Failure Recovery Guide as well as the JTA Programmers Guide.

What problems will this lead to?

You've had a failure. Until you provide a new XAResource, the transaction cannot complete. Locks and other resources will be retained.

Is there an inexpensive solution?

Since you're seeing this I'm assuming you've got more than one participant in the same transaction. If that's the case you needed XAResources in the first place, or you risk getting data consistency problems. You really only have two choices: a) go back to multiple one-phase resources and continue to run the risk of data corruption, b) fix the warning by adding in the necessary recovery classes.

I'd recommend b).

[jantzen]

Hi Mark, thank you for the detailed explanation, I appreciate it.

[marklittle]

I think I definitely need to add a wiki page around this, because it comes up enough.

[jantzen]

Hi Again,

I'm returning to this problem after a few months' hiatus. Has anyone created a wiki page or forum post with a how-to describing multiple one phase setup in JBoss 4.2?

Thanks

[jhalliday]

"jantzen" wrote:
Hi Again,

Has anyone created a wiki page or forum post with a how-to describing multiple one phase setup in JBoss 4.2?

Thanks

Is there something wrong with the one Mark mentioned back in July?
http://wiki.jboss.org/wiki/Wiki.jsp?page=Multiple1PC

[jantzen]

I've read that wiki page, along with the sections of the Transaction Manager manual that it references. None of these resources contain a simple example of how to accomplish the multiple 1 phase datasource.

I'm considering a premature upgrade to 4.2 due a bug in 4.0.5 (http://jira.jboss.com/jira/browse/JASSIST-28). Sorry to sound like a loser here, but I don't have the luxury of a week to bone up on the JTA internals and write a special implementation of com.arjuna.ats.jta.resources.LastResourceCommitOptimisation. If there's not a copy n'paste workaround here I've gotta look at other options.

[marklittle]

You don't have to "bone up" on anything. Just do what the documentation says:

"[set] com.arjuna.ats.jta.allowMultipleLastResources to true."

[jantzen]

Okay, thanks. I updated the Multiple1PC wiki page with a small clarification on how to do this.

But, now I'm seeing these errors:

11:47:51,510 ERROR [STDERR] Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException:
Cannot open connection

11:47:51,838 ERROR [STDERR] org.jboss.resource.connectionmanager.JBossLocalXAException: Trying to start a new tx when old is not complete! old: < 131075, 28, 26, 1-7f000001:880e:47375c0b:19a7f000001:880e:47375c0b:1ad >,
new < 131075, 28, 26,
1-7f000001:880e:47375c0b:1f07f000001:880e:47375c0b:1f7 >, flags 0

11:47:52,128 ERROR [JDBCExceptionReporter] Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 7f000001:880e:47375c0b:200 status:
ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable:
(javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 7f000001:880e:47375c0b:200 status: ActionStatus.ABORT_ONLY >))

Maybe if I explain what I'm trying to do you can point out my folly. I have a PostgreSQL database with multiple schemas. I want to read large sets of data from one schema, process the data in memory, write a subset of that data to the second schema, and update the first schema with a record that it has been imported. Here's a simple outline:

Postgres Database
------------------------------------------------------------------------------------------------
 Raw Source Data Schema | Processed Authoritative Data
 ---reads--> Source Data Processor App ---writes-->
 <--writes---

Large result sets for new data are read from Raw Source into memory using default "Required" JTA transaction.

JBoss application processes the data deciding what to do with it.

application chooses not to use the data, does nothing
application chooses to write the data to Authoritative

updates Authority in a "RequiresNew" JTA transaction
in the same transaction updates the original Raw Source record with the date on which it was processed. The source record is retrieved by id inside of this transaction, and the record as it exists in the dataset in step 1 is untouched.



I'm seeing exceptions now in all steps inside step 2. The same code works fine in 4.0.5. I'm using RequiresNew when writing authoritative records because on these large datasets the transaction gets to be gigantic and I want to keep it small and narrowly focused.

Any ideas why multiple one phase data sources fail in 4.2.1 even with the new property set? Or why I'd have transaction failures using XA transactions?

Thanks for your help.

[jantzen]

Oh now this is fun: every single property in that jbossjta-properties.xml uses YES or NO except for this: name="com.arjuna.ats.jta.allowMultipleLastResources" value="true".

Now it's working with multiple local datasources.

Mark, I'd still truly appreciate any light you could shed on why my use of transactions doesn't work with XA datasources. Thanks.

[marklittle]

"jantzen" wrote:
Oh now this is fun: every single property in that jbossjta-properties.xml uses YES or NO except for this: name="com.arjuna.ats.jta.allowMultipleLastResources" value="true".

It's even better than that: some of them use ON or OFF too ;-)

Now it's working with multiple local datasources.

Mark, I'd still truly appreciate any light you could shed on why my use of transactions doesn't work with XA datasources. Thanks.

It's difficult to tell for sure from the errors you posted, but it looks like something isn't terminating the transaction associated with a thread before that thread tries to start a new transaction. The fact the transaction is set to ABORT_ONLY probably points at something somewhere calling setRollbackOnly and assuming it disassociates the thread-to-transaction mapping, which it doesn't do.

[jantzen]

It's looking like one transaction, perhaps the read operation from the source data schema, is being marked for rollback, while the write operations later succeed, and the whole thing gets rolled back on the basis of a "mixed heuristic". I don't see any actual errors during execution though, so it makes no sense to me that any of the transactions would be rolled back.

What's the best way to determine the cause of the rollback? Enable trace logging in the arjuna package?

[marklittle]

"jantzen" wrote:
What's the best way to determine the cause of the rollback? Enable trace logging in the arjuna package?

Yes, that would be a good place to start.

[jantzen]

With debug logging on in "org.hibernate" and "com.arjuna" we were able to trace the call to setRollbackOnly to a class we're using to page over result sets. The method in question is this:

 private void getNextResults() {
 query.setFirstResult(tripCounter++ * maxResults);
 query.setMaxResults(maxResults);
 currentResults = query.getResultList();
 currentResultsIter = currentResults.iterator();
 }

"query" is a javax.persistence.Query object, and the method operates over the result set pulling out "maxResults" (ie 100) at a time. For some reason this operation encounters an error periodically inside of Query.getResultList(), which tries then to roll back the transaction.

This seems completely innocuous and there isn't an accompanying exception we can see. Could it be a timeout on the read operation? What could possibly be going wrong in iterating over a resultset?