recovery rollbacks a preparing TX

[dimonv]

Hi all,

I'm using JBoss TS 4.3.0GA. During my tests I set two breakpoints:
1. com.arjuna.ats.arjuna.coordinator.BasicAction.prepare (boolean reportHeuristics) line 2418 at the code

catch (IndexOutOfBoundsException e)
 {
 // end of list!

 --> ok = false;
 }

2. com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XAResource xares) at the line 955:

 if (!transactionLog((Xid) xids[j]))
 --> xares.rollback((Xid) xids[j]);
 else

In debug mode on a transaction I break the tx on the first breakpoint and wait for the second breakpoint. Then I resume the second breakpoint and the first and get an exception on commit:

XAResourceRecord.commit - xa error XAException.XAER_NOTA

In my opinion this is an issue. It has indeed a low probability and does not corrupt the data, but such behavior is not undesirable.

Is there any way to avoid this issue?

Thanks a lot.

[marklittle]

Maybe you could simplify your scenario a bit! Are you crashing your transaction during prepare and then expecting the recovery subsystem to commit it? If so, that isn't going to happen: there is no log written until AFTER prepare has completed.

[dimonv]

Thanks a lot for your replay.

I'm not crashing my transaction. I'm just simulating a simultaneous execution of two methods from JBoss TS.

[marklittle]

Sorry, you've still confused me. "simultaneous execution of two methods from JBossTS"? Huh? Why? What is your scenario? Don't quote code at this point. Just describe the scenario and your expectations. Then we can figure out if there's an issue or not.

[dimonv]

Sorry for confusion. It was indeed a bit bumpy, please let me try again.

In my scenario I don't kill any XA resources or JDBC-connections. This is a boring use case for a XA transaction: write access to Oracle database.
But the JBoss is running in the eclipse IDE in debug mode. I made two breakpoints as described above. The thread executing the use case is suspended on the first breakpoint so, that the TX has almost finished its prepare phase. Any when the second breakpoint suspends the recovery thread. If I resume the periodic recovery thread it rollbacks the current transaction.
It shows that the XA recovery can affect the transactions by rolling them back.

Please let me know if I could clarify.

[marklittle]

So let me try to rephrase this: you have two threads, t1 and t2. t1 is within prepare but not quite finished yet. t1 is blocked. t2 comes along and drives recovery. It enquires of the XAResourceRecovery instance to get any active transactions (XIDs). It then proceeds to roll back those that it believes are dead. It does this over a period of time, using multiple passes, backoff periods etc. to try to avoid rolling back anything that is actually in flight. Unfortunately you have blocked t1, so even if t2 waited until the heat death of the universe, t1 will not respond and let the transaction complete.

Therefore, the recovery thread correctly determines that the transaction has died (or at least it can't tell the difference any more) and decides to err on the safe side (always a good thing for a transaction service to do). It therefore rolls back the transaction. When you wake up t1 it finds it's no longer within the transaction and gets an appropriate XA error code.

Everything seems fine to me if that is a correct rephrasing of your scenario.

What would you have the TM do? Wait forever? As an industry we decided back in the 1960's that strict 2PC was not really a good thing and introduced heuristics. So although waiting forever is theoretically the correct thing to do, it's not applicable outside of academic scenarios.

[dimonv]

Hi Mark,

your rephrasing is correct.

What would you have the TM do? Wait forever?

Or course, wait forever is wrong; I would not expect such a behavior from a TM. But what I expect is a transaction timeout, which defines a period a transaction is considered to be alive and in progress. The recovery should check, how long is a transaction in progress and roll back only those transactions, whose timeout is expired.
At the moment exists a probability that a healthy transaction can be just rolled back unexpectedly by recovery. And this probability growths with the load.
The application requirements regarding the throughput can be quite high; up to thousands transactions per minute; and it is difficult for me to rate the impact of the recovery.
Is there really no hint for the recovery to look, how long is a transaction in progress, before it makes decision to roll it back?

[marklittle]

"dimonv" wrote:
Hi Mark,

your rephrasing is correct.

What would you have the TM do? Wait forever?

Or course, wait forever is wrong; I would not expect such a behavior from a TM. But what I expect is a transaction timeout, which defines a period a transaction is considered to be alive and in progress.

Yes, there is such a thing. I assume you've read the JTA specification, right?

Well the timeout only triggers for transactions that are not terminating. In your case the transaction is terminating so that timeout does not go off.

The recovery should check, how long is a transaction in progress and roll back only those transactions, whose timeout is expired.

Erm, no. That's what the timeout does in the JTA. But in your case what is happening is that recovery is running and, after quite a while, still thinks your transaction is dead so should be rolled back.

Timeouts are always going to be a probabilistic approach to failure suspicion. You cannot guarantee to detect a failure through timeouts, so everything that uses them eventually makes a heuristic choice.

I assume you are seeing the logging messages from recovery about going through phase 1, phase 2 etc? Also what timeout was associated with your blocked transaction?

At the moment exists a probability that a healthy transaction can be just rolled back unexpectedly by recovery. And this probability growths with the load.

Well you're not exactly running with a production setup, now are you? I don't know of any runtime that would purposefully pop up a debugger and stall an active thread in prepare!

The application requirements regarding the throughput can be quite high; up to thousands transactions per minute; and it is difficult for me to rate the impact of the recovery.

Well JBossTS has been running with recovery for over 10 years in many different production environments with a variety of loads. This has never been an issue. What I suggest you do is try to mock up a real-world test case that just thrashes the system and see if you get any of these rollbacks, because otherwise it's still very theoretical.

Is there really no hint for the recovery to look, how long is a transaction in progress, before it makes decision to roll it back?

Yes there is, but recovery is designed to have minimum impact on the running system. We optimise for the case where failures don't happen. If we started checking the lifetime of running transactions within recovery, that will have an impact on everyone else (timeouts are not directly associated with XIDs, unfortunately, so there's a lookup required which obviously has to be thread-safe). Then the question becomes: is this potential for rolling back a transaction that is active but just taking a while to complete better than any overhead imposed on dynamic lookups and additional tx pruning? At the moment the answer is pretty clear: yes.

I've created http://jira.jboss.com/jira/browse/JBTM-368 so you can vote on this. But if recovery is going through all of it's passes (phase 1, phase 2 etc.) before rolling back transactions, I don't see this issue as being as critical as some of the other things we need to do. Now if you come back to me and say "in this real-world mockup I get 50% transaction rollbacks as a direct result of recovery" we'll definitely reconsider.

[dimonv]

Thanks a lot for your explanations.

Timeouts are always going to be a probabilistic approach to failure suspicion. You cannot guarantee to detect a failure through timeouts, so everything that uses them eventually makes a heuristic choice.

Excuse me, but I didn't mean that the transaction timeout should be a key criteria for the recovery to decide if a transaction should be rolled back or not, but an additional one.

If we started checking the lifetime of running transactions within recovery, that will have an impact on everyone else

I thought that the recovery already does. It checks in the 2nd phase the state of the AtomicAction associated with the XAResource being recovered by calling XARecoveryModule.transactionLog(Xid). Unfortunately the AtomicAction, after it started prepare phase, has no entry in the ObjectStore. I've assumed that every status of AtomicAction is persisted to the OS. If so, then during prepare phase there would be an entry with ActionState.PREPARING in OS. If this would be the case the recovery would not roll the transaction back.
My other assumption was that if the timeout of an AtomicAction has expired the status of AtomicAction is then changed to ABORTED or ABORTING and updated in the OS.
The recovery could then check it in the phase 1 and remember it for the recovery in the phase 2 and there is no need for recovery to check the time out.
But may be I'm wrong with my assumptions.

I assume you are seeing the logging messages from recovery about going through phase 1, phase 2 etc?

The delay between phase 1 and phase 2 is default 10 seconds and transaction timeout is set to 300.

Well JBossTS has been running with recovery for over 10 years in many different production environments with a variety of loads. This has never been an issue.

That's probably what I supposed to hear.
Thanks a lot for your time and the interesting discussion.

[marklittle]

"dimonv" wrote:

Unfortunately the AtomicAction, after it started prepare phase, has no entry in the ObjectStore. I've assumed that every status of AtomicAction is persisted to the OS.

Not by default. We use presumed-abort, which means that we don't write to the log until after prepare has completed for all participants. You can override that, but that's a different story ;-)

If so, then during prepare phase there would be an entry with ActionState.PREPARING in OS. If this would be the case the recovery would not roll the transaction back.

See above. If we hit the log every time a transaction begins to prepare, then we lose the benefits of presumed-abort. If the recovery manager is running in-vm then it's possible for it to have access to the transactions directly, but that's not always the case.

My other assumption was that if the timeout of an AtomicAction has expired the status of AtomicAction is then changed to ABORTED or ABORTING and updated in the OS.

OK, I begin to see where there was confusion. As I said above, we don't hit the log that often.

The recovery could then check it in the phase 1 and remember it for the recovery in the phase 2 and there is no need for recovery to check the time out.
But may be I'm wrong with my assumptions.

Well you could certainly implement a TM that way. Presumed abort is just one of the ways you can go. There's presumed commit and presumed nothing, both of which make different assumptions and offer different trade-offs. But presumed abort is the most common approach.

Well JBossTS has been running with recovery for over 10 years in many different production environments with a variety of loads. This has never been an issue.

That's probably what I supposed to hear.

If that does change, we'd be sure to change things :-)

Thanks a lot for your time and the interesting discussion.

You too.