8 Replies Latest reply on Jul 3, 2006 9:02 AM by belaban

Improper handling of TXs in JBossCache?

manik Jul 3, 2006 7:44 AM

After a short conversation with Kevin Conner about the way we deal with 2PC transactions, I believe we have an incorrect implementation. From the email thread:

"Manik Surtani" wrote:
I have a resource, R, participating in a transaction, and registers a Synchronization S. When the transaction starts to commit, the TM calls S.beforeCompletion() which completes successfully. The TM then moves to the second phase and calls S.afterCompletion(). Now somewhere in the process of S.afterCompletion(), an exception occurs.

What would the 'correct' thing to do be at this stage:

a) initiate a rollback with tx.rollback() or tx.setRollbackOnly()?
b) throw an exception and let the TM deal with it?

I suspect it is b), but just wanted some clarification.

"Kevin Conner" wrote:

Neither is appropriate. The call to afterCompletion occurs after the transaction has finished, you no longer have a transaction context.

"Manik Surtani" wrote:

The reason why S.afterCompletion() may throw an exception is that this step involves replicating a commit call to remote caches in a cluster. At this stage, we may have a network hiccup which will cause this step to fail.

"Kevin Conner" wrote:

What you should really be doing is creating a resource that can take part in the 2PC protocol and replicate it as part of the protocol.

The other issue you will face is that calls to afterCompletion are not guaranteed to occur. If there was a failure after the intentions list had been logged then recovery would only restore the resources, not the synchronisations.

"Manik Surtani" wrote:

So are you suggesting that each instance in the cluster registers a synchronisation?

"Kevin Conner" wrote:

No, the opposite. :-)

What I am suggesting is that you use an XAResource instead of a synchronisation. This can then take part in the 2PC on the appropriate instance and, as part of the 2PC protocol, replicate the information across the cluster. It can also cause the transaction to fail should that be appropriate.

1. Re: Improper handling of TXs in JBossCache?

manik Jul 3, 2006 7:50 AM (in response to manik)

I presume this would then create a dependency on XA-compliant TMs (for synchronous replication anyway, which uses 2PC. Async replication uses 1PC)
Actions
2. Re: Improper handling of TXs in JBossCache?

manik Jul 3, 2006 7:55 AM (in response to manik)

And regarding using an XAResource rather than a Synchronization, I presume I'd still be wiring the replication code myself in XAResource.prepare(), bringing us back to the same problem of an exception that may occur in having to replicate a commit call in XAResource.commit().

Or have I misunderstood this? :-)
Actions
3. Re: Improper handling of TXs in JBossCache?

kconner Jul 3, 2006 8:27 AM (in response to manik)

presume this would then create a dependency on XA-compliant TMs

Any JTA compliant transaction manager will do
Actions
4. Re: Improper handling of TXs in JBossCache?

belaban Jul 3, 2006 8:28 AM (in response to manik)

Yes, I don't think using an XAResource would change anything here. In the original scenario, you *have* to be able to complete the COMMIT successfully once you voted for PREPARE. So, if a communication problem occurs sending the COMMIT decision to cluster member P, then P should be excluded from the cluster. Example: if P crashed between acking the PREPARE and receiving the COMMIT, we are fine, because when P is restarted, it will get the entire in-memory state from the cluster coordinator.
The difficult case is the following: what happens if there is a *temporary* situation which prevents us from sending the COMMIT message to P, but which does *not* exclude P from the cluster ? Retry COMMIT(P) until
- COMMIT(P) is successful or
- there is a view excluding P ?
I think the above mechanism would work quite nicely, because the fail-stop model of JGroups says that a member either successfully receives all messages, or it fails and is excluded from the group. So, COMMIT(P) would be either successfully delivered if there is only a temporary communication problem, or P would be excluded from the cluster group and therefore COMMIT(P) would not *have* to be delivered, assuming that we deliver messages only to non-faulty cluster members.
The issue here though is that we need to tackle the problem of a network partition and subsequent merge...
Actions
5. Re: Improper handling of TXs in JBossCache?

kconner Jul 3, 2006 8:35 AM (in response to manik)

I presume I'd still be wiring the replication code myself in XAResource.prepare(), bringing us back to the same problem of an exception that may occur in having to replicate a commit call in XAResource.commit()

The resource is still responsible for persisting the changes related to the transaction, however that is achieved. So yes, you will still have to manage the replication across the cluster.

How are you communicating with the other endpoints? If it is unicast then I would suggest creating a resource per endpoint.

This solution is different for two reasons though,

The commit occurs within the context of the transaction and can therefore notify the transaction manager of failure.
The commit will be guaranteed to happen in a recovery situation
Actions
6. Re: Improper handling of TXs in JBossCache?

kconner Jul 3, 2006 8:40 AM (in response to manik)

what happens if there is a *temporary* situation which prevents us from sending the COMMIT message to P, but which does *not* exclude P

What kind of situation are you thinking about?
Actions
7. Re: Improper handling of TXs in JBossCache?

galder.zamarreno Jul 3, 2006 8:59 AM (in response to manik)

I guess Bela might be refering to two nodes temporarily losing connectivity, but in the timeframe of the failure detection, the connectivity is restored and P is not excluded.

Failure detection in JGroups is based around FD/FD_SOCK plus VERIFY_SUSPECT.

I guess the transaction timeout would have to be bigger than the failure detection overall time.
Actions
8. Re: Improper handling of TXs in JBossCache?

belaban Jul 3, 2006 9:02 AM (in response to manik)

Here's one:
- a switch temporarily discards packets received on a given port (e.g. the IGMP snooping problem in certain routers, where routing tables are lost some times, and need to be reconstructed. During this time, all multicasts are simply discarded !).
- The COMMIT(P) call times out, throwing an exception (only if the COMMIT phase is synchronous)
- The switch resumes accepting packets on the given port
- P is not excluded, but the COMMIT(P) call failed

I think here a retry mechanism could solve this problem
Actions

Go to original post