Can resource delisting prevent participation in commit processing?

[dmlloyd]

With the remote JTA protocol we use with WildFly EJB, it may come to pass that multiple remote subordinates have been enlisted in the transaction, yet only one of them actually participated in it.  During the period where participation is in doubt, we enlist the subordinate XAResource to ensure that in the event of a crash, recovery can work correctly.

A problem arises during prepare processing - while we may know that in a given specific case only one resource will return XA_OK from prepare (the rest returning XA_RDONLY), it's only a matter of chance as to whether that particular resource will end up last in the list (and thus be eligible for 1PC optimization).  It would be ideal if we could indicate, ahead of time, that the transaction branch did not do any work.

My question is, can we do so using Transaction.delistResource() somehow?  Reading the code, it looked to me like all resources which were ever enlisted all participate in commit processing, but I'm not sure if I had misread the code.  If not by delistment, is there some other (preferably standard) way to indicate that a resource had done no work, before prepare begins?

[ochaloup]

I did some investigation as I consider this question related to [WFLY-10258] Prevent enlisting additional resources on EJB caller side in case of server side @RequiresNew.

In case of the ejb remoting XAResource there is already implemented the functionality where the delisting happens in beforeCompletion callback (wildfly-transaction-client/XAOutflowedResources.java at master · wildfly/wildfly-transaction-client · GitHub ). If I focus on this particular case where ejb remoting XAResource is used (and I will leave aside some generic solution) it seems to me there could be an issue in the Transaction (Java(TM) EE 7 Specification APIs) #delist call in the Narayana implementation.

I can see that the wfly transactional client delists itself from the current transaction in beforeCompletion calllback when it knows there was no work done (https://github.com/wildfly/wildfly-transaction-client/blob/master/src/main/java/org/wildfly/transaction/client/XAOutflowedResources.java#L79). That's good but delisting at the Narayana side (https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/transaction/arjunacore/TransactionImple.java#L904) seems not removing the xa resource from the list (https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes/com/arjuna/ats/arjuna/coordinator/BasicAction.java#L3719) which defines if 1PC is applied.

What do you think tomjenkinson about my observation?

[mmusgrov]

During prepare processing if a resource returns XA_RDONLY then it is removed from the prepared list so in your example the only resource to be committed is the one that returned XA_OK during the prepare phase. In other words the processing of participants is deterministic.

[tomjenkinson]

As Mike's response confirms returning RDONLY should mean it won't be called again, is that not what is being observed?

[ochaloup]

tomjenkinson mmusgrov : for more details and reasoning please refer the comments at WFLY-10258

the observed point is that we are not able to decide about 1PC vs. 2PC until time the prepare is called.

The scenario is: we have 2 resources - the first one is e.g. database and the second one is the ejb remoting XAResource. The 2PC begin and prepare is called on the database. Then the ejb remoting returns READ_ONLY at prepare call. Next the commit is called on database.

But there is no reason to call prepare on the database as the transaction contains 2 resources where only one (database) does some real work where the other one (ejb remoting) has nothing to commit. We would be fine to run just the commit (1PC) for the database.

The optimization here is to ensure we would find how many resources really need to commit "before" the whole 2PC machinery is started.

[mmusgrov]

Prepare is the trigger point for the resource to decide what if any work was performed. Changing the semantics around enlist or end would require buy in from other resource managers so it would be a non-starter.

If this is a JBR only question then we could order JBR resources before other resources.

[dmlloyd]

mmusgrov  wrote:

Prepare is the trigger point for the resource to decide what if any work was performed.

I was afraid of that.

mmusgrov  wrote:

If this is a JBR only question then we could order JBR resources before other resources.

I can already do that, but unfortunately if there are multiple JBR resources then we're back in the original situation.

mmusgrov  wrote:

Changing the semantics around enlist or end would require buy in from other resource managers so it would be a non-starter.

I disagree that it would be a non-starter.  Here's a way it could be done (even in spec) in a compatible fashion.

First, enhance the XAResource interface adding the following default method:

/**
 * (Copy description from #end)  The return value specifies whether any work was completed by the resource during this assoication.
 *
 * @return one of XA_RDONLY or XA_OK
 */
default int endWithResult(Xid xid, int flags) {
    end(xid, flags);
    return XA_OK;
}

(I only added abbreviated JavaDoc because the Mojo editor is being terrible.)

The idea would be that resources which support this optimization can override the endWithResult method as well as the end method.  Resources which do not support it will always return XA_OK from the default method.  Transaction managers which support the optimization will call endWithResult, and transaction managers which do not will continue to call end as they do today.

If the transaction manager supports endWithResult, it would note whether each resource returned XA_OK and XA_RDONLY, and would make a decision based on that.  If XA_RDONLY was returned, it could (for example) prepare those resources first, or roll them back and exclude them from prepare altogether.  Then prepare could deal with a shorter list of resources and the benefits that may derived therefrom.

Of course we could also do this kind of thing without a spec change, using a proprietary subinterface of XAResource, but the benefit would be more general if the spec were updated.

[tomjenkinson]

I am not entirely certain why JBR can't just cache the knowledge that it did not perform any work as some internal value that it can return to the XAResource it registered with the TransactionManager so that it can return RDONLY without an additional RPC?

Regarding ordering though, I think there are several XAR registered (from JBR) for effectively the same resource (a subordinate server) and ideally only one of these (the one that did work) would be asked to prepare. This does lead to the question of whether JBR might try to only register one XAR per subordinate server and somehow maintain some internal state for the various branches itself perhaps?

[dmlloyd]

tomjenkinson  wrote:

I am not entirely certain why JBR can't just cache the knowledge that it did not perform any work as some internal value that it can return to the XAResource it registered with the TransactionManager so that it can return RDONLY without an additional RPC?

Regarding ordering though, I think there are several XAR registered (from JBR) for effectively the same resource (a subordinate server) and ideally only one of these (the one that did work) would be asked to prepare. This does lead to the question of whether JBR might try to only register one XAR per subordinate server and somehow maintain some internal state for the various branches itself perhaps?

We already do return RDONLY, however that is not sufficient, particularly when there are multiple resources or multiple subordinate servers involved.

The code *should* only be registering one XAR per subordinate, though we do presently aggressively delist when we can (this was in the hopes that the TM would delete our resource, but this appears not to be the case).  So perhaps it is being observed that the same XAR instance is being delisted and re-enlisted many times.  Given that the hypothetical delist optimization doesn't actually work in practice, we could probably delete that code anyway as it's not doing anything beneficial for us.

[tomjenkinson]

I assume you are referring to an update to the JTA spec rather than the XA spec. But in fact the modification you are proposing would be more appropriate in the XA specification itself in my opinion.

A thread can be associated and disassociated several times with the same branch. I assume the XAR would be responsible to maintain whether it is RDONLY or not. e.g. this it not be valid to change back to RDONLY from XA_OK? Or if it was OK to do so, there would be no expectation that the transaction manager to track the state except the last call:

xar.start(xid1)

xar.end(xid1) -> RDONLY

xar.start(xid1)

xar.end(xid1) -> XA_OK

xar.start(xid1)

xar.end(xid1) -> RDONLY

But what I am wondering is whether there would be a win in this is as we know that one of these resources did do some work? For example, you have two servers and propagate to several subordinates on server2 from server1 and so server1 ends up with three XAR and as you say only one of those has updates. Is there really any problem to call xar1.prepare() -> RDONLY< xar2.prepare() -> XA_OK, xar3.prepare() -> RDONLY as the RDONLY ones could be using information you have maintained from the remote calls to not need to propagate that to server2 to know it was RDONLY. You would of course still have the 2PC for the the subordinate that did do some work and I think it is this you are trying to avoid. The thing I could think of is that you somehow check internally whether you have an existing XAR for a server2 and in that case don't register the multiple XAR at server1 for the various subordinates but try to maintain that state internally in your XAR perhaps?

I do want to make the point that what you asking for does make sense conceptually. The more recent protocols we implement such as LRA and REST-AT both allow a participant to LEAVE a transaction at any time - unfortunately XA does not allow that.

[dmlloyd]

tomjenkinson  wrote:

I assume you are referring to an update to the JTA spec rather than the XA spec. But in fact the modification you are proposing would be more appropriate in the XA specification itself in my opinion.

I would say both, and I would say separately, particularly given that the nature of such an optimization is compatible but that compatibility might be subject specifically to the programming language involved.  I don't think there's a good reason to lock JTA to XA just for the sake of having done so, especially given that (as far as I understand it) there already are specific differences between those two specifications.

tomjenkinson  wrote:

I assume you are referring to an update to the JTA spec rather than the XA spec. But in fact the modification you are proposing would be more appropriate in the XA specification itself in my opinion.

A thread can be associated and disassociated several times with the same branch. I assume the XAR would be responsible to maintain whether it is RDONLY or not. e.g. this it not be valid to change back to RDONLY from XA_OK? Or if it was OK to do so, there would be no expectation that the transaction manager to track the state except the last call:

xar.start(xid1)

xar.end(xid1) -> RDONLY

xar.start(xid1)

xar.end(xid1) -> XA_OK

xar.start(xid1)

xar.end(xid1) -> RDONLY

But what I am wondering is whether there would be a win in this is as we know that one of these resources did do some work? For example, you have two servers and propagate to several subordinates on server2 from server1 and so server1 ends up with three XAR and as you say only one of those has updates. Is there really any problem to call xar1.prepare() -> RDONLY< xar2.prepare() -> XA_OK, xar3.prepare() -> RDONLY as the RDONLY ones could be using information you have maintained from the remote calls to not need to propagate that to server2 to know it was RDONLY. You would of course still have the 2PC for the the subordinate that did do some work and I think it is this you are trying to avoid. The thing I could think of is that you somehow check internally whether you have an existing XAR for a server2 and in that case don't register the multiple XAR at server1 for the various subordinates but try to maintain that state internally in your XAR perhaps?

I do want to make the point that what you asking for does make sense conceptually. The more recent protocols we implement such as LRA and REST-AT both allow a participant to LEAVE a transaction at any time - unfortunately XA does not allow that.

If you're specifically concerned with Remoting, we do in fact track our state such that we would never return RDONLY after returning OK, were an optimization like this to be in place.

In the general case, as an RM maintainer I would want the TM to remember if I ever returned XA_OK so that I don't have to.  Given the N:1 nature of RM:TM, it's better to put bookkeeping/boilerplate work on the "1" side (TM) rather than the "N" side (RMs) whenever possible.  Specifically in Narayana, I'd imagine that such information would logically map to a "set-once" field on com.arjuna.ats.internal.jta.xa.TxInfo which remains false until/unless endWithResult returns XA_OK at which point it would toggle to true and stay there.  Then the prepare processing can check the field.

[ochaloup]

If you don't mind I would take a side step to the discussion and ask:

is it correct if Transaction.delistResource does not delist resouce from the handling of the 1PC/2PC decisioning. Should not be that considered as a bug?

[tomjenkinson]

Hi Ondra, delistResource is just a way to register the resource from the transaction so you can do work outside the transaction - it doesn't actually leave the transaction.

[tomjenkinson]

It seems to me that the best solution would just be to add a "leave(XAResource)" operation that you can call on the TM during before completion or something. But mmusgrov can you give a little context around the similar operation in REST-AT?

[dmlloyd]

tomjenkinson  wrote:

It seems to me that the best solution would just be to add a "leave(XAResource)" operation that you can call on the TM during before completion or something. But mmusgrov  can you give a little context around the similar operation in REST-AT?

Sure, that works too (and incidentally would require very little change on our part).