JBTM-2124: Add orphan detection for JTS interposition mode

[ochaloup]

I would like to sketch here my current effort to fix https://issues.jboss.org/browse/JBTM-2124 : Add orphan detection for JTS interposition mode.

The point is to gather feedback about the feature fix and about the way how it's implemented.

The PR currently resides at: https://github.com/jbosstm/narayana/pull/1174

For the basic overview where I started, please, read the comments on the issue

https://issues.jboss.org/browse/JBTM-2124?focusedCommentId=13020162#comment-13020162

https://issues.jboss.org/browse/JBTM-2124?focusedCommentId=13029295#comment-13029295

Now, what's the issue and what I'm trying to achieve?

The orphan detection in Narayana is a term used for the process during periodic recovery. If you imagine having a transaction consisting of two resources. Business logic does some work (e.g. inserting data to databases) on those two resources and then it's time for transaction manager to finish the work in an atomic way - commit both resources as a single unit of work. At that time two-phase commit processing comes to work. The transaction manager starts with prepare phase and asks resource one and two to prepare. Each of the resources response 'ok' and make a durable record of its local transaction being prepared to its own transaction log. The prepare phase is now in the end and transaction manager would persist information about the transaction being prepared to its transaction log store. If it succeeds then the transaction is prepared for commit - either immediately in the next step of 2PC or later by periodic recovery process if some failure occurs. But let's imagine that the underlying system crashes before transaction manager durably persist information that the transaction was successfully prepared.

When the transaction manager system is restarted the transaction manager has no idea that there a transaction but resources have locked their data and their local transaction log contains info about the prepared local transaction. Recovery process asks the both resources and they provide that info to the transaction manager. Now the transaction manager should command to roll-back the local transaction on resources.

In Narayana it's concern of orphan filters (XAResourceOrphanFilter). Those are "strategies" applied on in-doubt transactions coming from resources which the Narayana's transaction log is not aware of. The filters vote what should be done with such in-doubt transaction. One of them is JTANodeNameXAResourceOrphanFilter which checks the Xid of the in-doubt transaction, unpack node name (identifier of the transaction manager who creates (prepared) the transaction) and if it's present and it's equal to node identifier of transaction manager running the recovery it votes for rollback. That way the periodic recovery decides to roll-back such in-doubt txn.

This behavior is already implemented for JTA but it misses for JTS. The point here is to mimic the JTA implementation for JTS. My change has two parts (I would say).

First, is enabling handling of NodeName Orphan filter for JTS. Currently, there is format check which permits the rollback being voted only for JTA transaction.

Second, is packing (saving) node name to ServerTransaction implementation where such handling is not provided so far. If the node name is saved to Xid the orphan filter can benefit from it and decide voting to roll back.

[mmusgrov]

On the JIRA tomjenkinson suggested a similar fix to the one he applied for JBR. But he then went on to say    

We need to consider whether this can work for both interposition and context propagation and whether there are caveats.

And nothing in the rest of JIRA indicates whether the same fix has addresses either of those two concerns. I must confess that I do not fully understand his concerns but until Tom has his answers to both questions I would advise postponing the fix.

[ochaloup]

mmusgrov  I did understand from Mark's last answer that it should be fine and is good to proceed with it. But that's why this thread was created to discuss possible issues. Do you have some specific idea what could go wrong? I fully admitting I do not have such good understanding of the topic to raise them myself. Thanks.

[tomjenkinson]

jhalliday wrote the original version of orphan detection - perhaps he would have some input as to why JTS was left at that time?

If you look at the orphan filters they expect to find the transaction log in the local object store. For JTS in EAP I expect this is true, special note would need to be made when the resources are potentially accessed in different nodes then the right node ID is looked for in the log (this is usually the case).

You could add a unit test that shows both interposition and context prop and that would help verify it works for both (assuming a simulation of remote coordinators with non-shared object store is put in place (like SimpleIsolatedServers test))

[jhalliday]

The original implementation (seven years ago!) was focused on providing the most benefit with the least effort. The JTS had relatively few users and we had bigger problems competing for our attention. As I recall I didn't address even JTA interposition (i.e. JCA transaction inflow) for the same reason.

Interposition poses some tricky issues for orphan detection. These are somewhat different in JTA and JTS (really jtax) due to the way transactions are identified.

Basic top level JTA orphan detection is built on encoding the node name of the coordinator into the Xid's gtrid field. That poses a problem for JTA interposition which, due to deficiencies in the JCA spec, forces the subordinate coordinator to use the parent's Xid verbatim. As a result the subordinate won't clean up orphans, since it considers them to belong to the parent. The parent may not clean them up since it doesn't necessarily connect directly to the resource manager. There is one loophole we can exploit: where the parent and child are both narayana, then the propagated Xid can be parsed and hacked around a bit, since we know its internal format. Thus unlike the general case, we can change the bqual, using it to encode the node info that previously lived in the gtrid.

JTS (jtax) is a little different, since the transaction has dual identities: the JTS one the transaction managers use between themselves and the XA one the resource managers use. Context propagation in JTS passes a list of the ancestor transaction references. A receiving coordinator usually adds a new link to that list, i.e. creates a distinct (sub)transaction identity Uid by interposition, though IIRC it's possible to have it masquerade as the parent. Or rather, it's possible to have the receiving node register the resource directly with the parent rather than interposing a local coordinator, since JTS resources are network endpoints in their own right, unlike the JTA case.  Assuming interposition, the subordinate's effective Uid is used for Xid creation. Note that the context propagation doesn't include the parent's node name since it is the JTS tx id rather than the Xid that's being passed around. As a result, Xids as seen by the resource managers, won't be considered to belong to the same transaction, since the parent and child coordinators will base their Xids on different Uids.

On the up side, that means JTS subordinates can use orphan detection as though they were top level, as least as far as node names are concerned. It's been a while but IIRC the jtax code shares the same Xid construction functions as the jta, so it's probably already putting the node name into the gtrid?  The additional bit to make it work is not really the name filter, it's the ownership filters (live and store). You can't rollback prepared branches unless they really are orphans, so you have to look for the owning tx. The existing JTA filter only looks for records in the JTA part of the store. Since JTS tx have a different type, they will be in a different place. And to make life even more fun, top-level and subordinates ones will be separate. Thus in JTS mode for any orphan, you have to check both locations, since nothing in the Xid itself will tell you which type it belongs to. Arguably it may be desirable to change the JTS version of Xid to encode that info, though that makes the shared Xid building/parsing code more complex. OTOH the JTA will have the same problem anyhow? I should probably read the code before suggesting designs, it's been a while. But basically: ensure the node name is getting into the Xid (it probably already is), ensure the existing name filter can read it (it probably can), add ownership filter(s) as necessary.  Not sure why you want to pack the node name to ServerTransaction, it's read from the config file at startup?

[ochaloup]

Hi jhalliday,

thank you for your answers! I reworked my PR a little bit but mostly in details right now. I would like to ask you for a few more notes for me being able to continue.

Regarding the effort and the benefit. I understand your point. Based on discussion at JBTM-2124 it seemed to me that this could not be so problematic if the JTA behavior would be mimicked. Plus when talking with Tom it seems to be an appropriate task for me getting more familiar with the codebase. Even the JTS is not used so heavily I think the automatic orphan handling is beneficial to have been implemented. Nevertheless, I could be wrong in these assumptions :-)

I would like to sum up my understanding here to ensure that I'm on the same page as you are. In the context of JTA and in the context of the 'loophole' where Xid is maintained by Narayana:

When transaction is propagated from one instance of Narayana to other instance of Narayana (let's say from "narayana1" to "narayana2") then a participant is enlisted to subordinate (propagated) transaction the participant's xid contains _node name_ being set to "narayana1" and contain record _subordinate node name_ being set to "narayana2". When a crash happens the recovery orphan filter checks the content of _node name_, the subordinate orphan filter checks the content of _subordinate node name_. If node name matches in Xid matches the active one of periodic recovery it votes to *rollback*.
The recovery process is run on each node independently thus the recovery manager on each node uses its own orphan filter check.
All this process works when Xid is maintained and known to us. For ensuring it the orphan filter verifies if Xid's _format_ refers to the Xid created by Narayana.

  Referring to your suggestions

1. I can confirm that the _jtax_ uses the same Xid construction as JTA.
2. ensure the node name is getting into the Xid (it probably already is): I think it's, but I would no say that being used during recovery
3. ensure the existing name filter can read it (it probably can): I think the existing name filter could be able to read it but some hooking would be necessary. My change of having the node name being encoded as an additional field as JTA has it makes the existing name filter reading it.
4. add ownership filter(s) as necessary: I do miss this part of puzzle as I need to study how to create one
5. not sure why you want to pack the node name to ServerTransaction, it's read from the config file at startup?: my idea was to add the node name information to imported transaction too. I wanted to follow the JTA pattern where node name is encoded as special Xid field

Now, please, let me ask you for more details.

I understand from your reaction that changing the `ServerTransaction` in the way to encode nodename there is not what you would expect. From my point of understanding it's the place where I want to encode the node name to Xid for get it working in similar fashion as JTA does. Would you be so kind to elaborate on this?

On adding the ownership filter - I understand it in way that I need to implement similar handling as JTA orphan filters do - https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/JTAActionStatusServiceXAResourceOrphanFilter.java
https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/JTATransactionLogXAResourceOrphanFilter.java
Am I getting it right?

Thanks again!
Ondra

[tomjenkinson]

Just to say that the subordinate node name is encoded in the bqual in the JTA case:

narayana/SubordinateAtomicAction.java at master · jbosstm/narayana · GitHub

Also, the orphan filter that checks (on narayana2 say) whether there is a transaction log for the subordinate is the following:

https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/SubordinateJTAXAResourceOrphanFilter.java

[jhalliday]

> I can confirm that the _jtax_ uses the same Xid construction as JTA.

I hope you're wrong. The modules should have different FORMAT_ID values, or you can't tell which modules owns the Xid, which means you can't tell which ownership filters are relevant. The modules may use the same function to assemble the Xid components, but they should not use the same values in that function. Same construction *function*, but different construction.

> not sure why you want to pack the node name to ServerTransaction, it's read from the config file at startup?: my idea was to add the node name information to imported transaction too. I wanted to follow the JTA pattern where node name is encoded as special Xid field

right, but you can copy it into the Xid directly from the runtime config, you don't also need to store it independently in the transaction structure. The objectstore entry for the tx already contains the XAResourceRecords, which contain the Xid, which contains the node name. Changing the on-disk representation of the transaction just makes life difficult.

Ownership filters are tricky because you have to look in several places. There may be a live tx with no record on disk (the JTA uses the action status service filter for this case) and there may be a record on disk in one of (at least) two places: top level tx or subordinate tx. Arranging the filters so they perform all the necessary checks and don't tread on eachother's toes takes a bit of thought. It's actually the non-interposed case that worries me most, since it's most different to the JTA pattern. You don't have a local tx record in the store at all since you don't have a local tx, just a local Resource. That never happens in JTA, since JTA resources are not network endpoints and must be contained in a local tx. So the JTS probably needs filters similar to each in the JTA, but also one more.

[tomjenkinson]

Jonathan Halliday wrote:

> I can confirm that the _jtax_ uses the same Xid construction as JTA.

I hope you're wrong. The modules should have different FORMAT_ID values, or you can't tell which modules owns the Xid, which means you can't tell which ownership filters are relevant. The modules may use the same function to assemble the Xid components, but they should not use the same values in that function. Same construction *function*, but different construction.

Hi Ondra, JTS is (or should be) using narayana/XidUtils.java at master · jbosstm/narayana · GitHub

[tomjenkinson]

You would also need to consider orphan detection for JTS where we have an imported JCA transaction as that does not allow us to mutate the Xid so we can't do a check for orphaned XAResource

[ochaloup]

I see. Just I think that the imported JCA transaction should be left intact by definition.

There should not be applied any orphan detection strategy.

I think  it's similar to a prepared in-doubt transaction on DB side - they will be left forever prepared until somebody from outside (TM or a user) explicitly commands to be rolled-back/committed.

Is there something I haven't realized? Thanks.

[tomjenkinson]

I am thinking of an orphan XAResource that Narayana knows about but the external EIS doesn't.

i.e. EIS imports TX to WFLY. WFLY enlists an XAResource. EIS asks Narayana to prepare, we prepare XAResource then crash (the XAR is the orphan) before writing our log.

[ochaloup]

All right, I got it. Thanks for the explanation. I will be aware of it.

[ochaloup]

Hello,

recently I've started to work on the JTS interposition feature (JBTM-2124) after some time. I've got a new set of changes here [JBTM-2124] orphan detection for JTS · ochaloup/narayana@571f62e · GitHub.

I'm still using the "old" style of saving subordinate name to ServerTransaction I would like to ask jhalliday would you be so kind and elaborate on your quote (potentially with some codebase links)?

you can copy it into the Xid directly from the runtime config, you don't also need to store it independently in the transaction structure. The objectstore entry for the tx already contains the XAResourceRecords, which contain the Xid, which contains the node name. Changing the on-disk representation of the transaction just makes life difficult.

For example I'm kind of unsure what you meant that objectstore entry for the tx already contains the XAResourceRecords etc. Thanks in advance.

The other changes are part of reusing JTA orphan filters for the JTS. As I was researching around it I think they are capable to sort out the issues mentioned through this discussions and short discussion at Newcastle on f2f meeting last year (check existence of running transaction, check if matches subordinate name if it's subordinate transaction etc.).

---------------------------------------------------------------------------------------------------------------------------------------------------------------

As a second point I would like summarize the state of the current code base. It could help for further discussion maybe.

This part is not directly connected to the question above of the new PoC for JBTM-2124.

The list of the active jta filters and their functionality. This is just and overview to summarized the state.

(stored at https://github.com/jbosstm/narayana/tree/5.8.1.Final/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore)

• JTATransactionLogXAResourceOrphanFilter
  • processing JTA transactions only
  • ensures leaving alone transactions which have got an existing record entry in the object store (under type AtomicAction)
• JTANodeNameXAResourceOrphanFilter
  • processing JTA transactions only
  • checks if node name inscribed into the Xid is one of the nodes the TM is responsible for,  aka. filter checks if such node name is part of the JTAEnvironmentBean.xaRecoveryNodes array
• SubordinateJTAXAResourceOrphanFilter
  • processing any transactin (no format id check)
  • checks if subordinate(!) node name inscribed into the Xid is one of the nodes the TM is responsible for, aka. the name is part of the JTAEnvironmentBean.xaRecoveryNodes array
  • ensures leaving alone transactions which have got an existing record entry in the object store (under type SubordinateAtomicAction)
• SubordinationManagerXAResourceOrphanFilter
  • checks if subordinate(!) node name inscribed into the Xid is one of the nodes the TM is responsible for (duplication to the SubordinateJTAXAResourceOrphanFilter)
  • checks if transaction is not imported, if it's the imported one then it will be left alone
• JTAActionStatusServiceXAResourceOrphanFilter
  • checks if node name inscribed into the Xid is one of the nodes the TM is responsible for (duplication to the JTANodeNameXAResourceOrphanFilter)
  • vetos rollback for xids which have an in-flight transaction (using ActionStatusService)

plus my notes to the JTA vs. JTS hierarchy of transaction types (maybe usable for further discussion)

JTA

com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple

   -> com.arjuna.ats.internal.jta.transaction.arjunacore.subordinate.TransactionImple

      -> com.arjuna.ats.internal.jta.transaction.arjunacore.subordinate.jca.TransactionImple [com.arjuna.ats.internal.jta.transaction.arjunacore.jca.SubordinateTransaction]

         #getParentNodeName -> com.arjuna.ats.internal.jta.transaction.arjunacore.subordinate.jca.SubordinateAtomicAction

com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple [com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporter]

    #getInflightXids

com.arjuna.ats.internal.jta.transaction.arjunacore.jca.XATerminatorImple

    # getXidsToRecoverForParentNode

Hierarchy of the transction types in for JTA

(common com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator <- BasicAction <- StateManager)

com.arjuna.ats.arjuna.AtomicAction

   -> com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction

     -> com.arjuna.ats.internal.jta.transaction.arjunacore.subordinate.SubordinateAtomicAction

        -> com.arjuna.ats.internal.jta.transaction.arjunacore.subordinate.jca.SubordinateAtomicAction

JTS

com.arjuna.ats.internal.jta.transaction.jts.TransactionImple

   -> com.arjuna.ats.internal.jta.transaction.jts.subordinate.TransactionImple

      -> com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.TransactionImple [com.arjuna.ats.internal.jta.transaction.arjunacore.jca.SubordinateTransaction]

        #getParentNodeName -> com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.SubordinateAtomicTransaction

com.arjuna.ats.internal.jta.transaction.jts.jca.TransactionImporterImple [com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporter]

   #getInflightXids (not existence)

com.arjuna.ats.internal.jta.transaction.jts.jca.XATerminatorImple

   # getXidsToRecoverForParentNode (TODO)

Hierarchy of the transctio types in for JTS

(common com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator <- BasicAction <- StateManager)

com.arjuna.ats.internal.jts.orbspecific.coordinator.ArjunaTransactionImple

  -> com.arjuna.ats.internal.jts.orbspecific.interposition.coordinator.ServerTransaction

     -> com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.coordinator.ServerTransaction