TMFAIL vs. Xid

[jesper.pedersen]

Hi,

I'm looking at a scenario where the Enterprise Information System is expecting that it is notified with TMFAIL on the connection that actually had the error occur on it. Currently IronJacamar just "disassociates" (makes sure that afterCompletion is "cancelled") the transaction with the connection listener, and kills it afterwards.

The scenario basically looks like

ut.begin()
   xar = c.getXAResource();
   tx.enlistResource(xar);
   ...
   tx.delistResource(xar, TMFAIL);

The same connection listener will only be enlisted once per transaction, so there are no multiple Xid's associated with it (unless Narayana interleaving does something in that area).

However, in the above case Narayana allows for a ut.commit() - e.g.

ut.begin();
   xar = c.getXAResource();
   tx.enlistResource(xar);
   ...
   tx.delistResource(xar, TMFAIL);
ut.commit();

TMFAIL marks the transaction branch as rollback, but that doesn't seem to be propagated to the overall status of the transaction in this case.

So, does Narayana keep track of the status for each XAResource <-> Xid, and decides the outcome based on the last Xid "status" for each XAResource ?

I have added the TMFAIL on failure interaction here - Send TMFAIL and do setRollbackOnly before connection is killed · jesperpedersen/ironjacamar@76a6650 · GitHub - and explicit set the MARK_FOR_ROLLBACK here - Send TMFAIL and do setRollbackOnly before connection is killed · jesperpedersen/ironjacamar@76a6650 · GitHub as DELIST_RESOURCE isn't necessary enabled. Doing tx.setRollbackOnly() should be safe, as we know the connection listener had a fatal error occur on it, and it will be killed shortly after, e.g. in the method calling haltCatchFire().

I think the above case should do a RollbackException, but I may have overlooked some hidden details The XA spec seems to be most clear about this:

"

TMFAIL

The portion of work has failed. A resource manager might choose to mark a

transaction branch as rollback-only at this point. In fact, a transaction manager

does so for the global transaction. If a resource manager chooses to do so also,

xa_end() returns one of the [XA_RB∗] values. TMFAIL cannot be used in

conjunction with either TMSUSPEND or TMSUCCESS.

"

Thoughts on the matter would be great !

[tomjenkinson]

Narayana doesn't actually call setRollbackOnly as the only scenarios where it elects to use TMFAIL of its own accord are in situations where a transaction is already being aborted and that resource has not been delisted by or on behalf of the client application yet. We do call setRollbackOnly if the xa_end(TMSUCCESS) fails, for example: narayana/TransactionImple.java at master · jbosstm/narayana · GitHub

More generally, in determining whether it was the requirement of the transaction manager to also call setRollbackOnly (or indeed trigger an automatic rollback) as a result of someone calling delistResource(TMFAIL) we have to look at both the JTA and XA specifications. The text you quote does suggest it the responsibility of the TM to call setRO (or abort) if it elects to TMFAIL a branch unilaterally, when read in the context of JTA you can also see a picture where TMFAIL != setRollbackOnly. For example:

* JTA Javadocs: http://docs.oracle.com/javaee/7/api/javax/transaction/xa/XAResource.html#end(javax.transaction.xa.Xid, int) - "If TMFAIL is specified, the portion of work has failed. The resource manager may mark the transaction as rollback-only" - by referring to the resource manager being able to call setRO this indicates that it is not always the case that TMFAIL corresponds to an abort scenario.

* JTA specification delistResource - "TMFAIL - the resource is dissociated because part of the work has failed", i.e. only part of the work has failed, this does not mean that the whole transaction must abort as an XA transaction can have multiple branches actives in the scope of the same transaction and the failure of one of them should not necessarily cause the entire transaction to fail or be rolled back.

I see from your code you are calling setRollbackOnly when you want a transaction to rollback and are setting TMFAIL on the XAR - from what I can understand of your requirement this should be sufficient. The reason you are able to call setRO in your scenario is you are aware of the scenario causing haltCatchFire to be executed and so setRO makes sense for you whereas in some situations users may wish to use TMFAIL without setRO being executed and that is why Narayana can't call the setRO()

[jesper.pedersen]

My point was that Narayana doesn't track the "end" state of its Xid.

This scenario,

ut.begin();
  XAResource xar = c.getXAResource();
  tx.enlistResource(xar);
  tx.delistResource(xar, TMFAIL);
  tx.enlistResource(xar);
  tx.delistResource(xar, TMSUCCESS);
ut.commit();

should not result in a RollbackException, since the last Xid associated with the XAResource resulted in a success state.

However,

ut.begin();
  XAResource xar1 = c1.getXAResource();
  XAResource xar2 = c2.getXAResource();
  tx.enlistResource(xar1);
  tx.enlistResource(xar2);
  tx.delistResource(xar1, TMFAIL);
  tx.delistResource(xar2, TMSUCCES);
ut.commit();

should.

The setRO() call is to make sure that the transaction manager actually does a rollback, since ironJacamar can run w/o calling delistResource() - since it has an impact on how the transaction manager must sort the Synchronization instances.

A resource manager - at least in a JCA context - can't call setRO(), since there is no access to the actual transaction in which it is enlisted. Of course resource adapters may use hacks to get around that, but it is out of spec. So, the only way to tell the transaction manager that something is wrong is through the exception states associated with its XAResource implementation.

In this precise case we know that it is safe to call setRO(), since the associated XAResource will never be enlisted again - due to L6 in the "However" example. The transaction manager can choose to do a rollback due to other cases, like an exception in L7 (if L6 was TMSUCCESS).

So, I think it boils down to what "not necessarily" means in the above examples, and how it related to how the client should act - mandatory to call setRO(), or rely on the TM to do it.

[marklittle]

As Tom already pointed out, it would be inappropriate to set the transaction to rollback with xar1 in your example above. We have no way of knowing if there are other transaction branches proceeding along quite nicely (in this JVM or elsewhere) and having one RM force the rollback on behalf of all of them makes no sense whatsoever. Remember the structure of the Xid contains a global id and a branch id?

[marklittle]

You say "A resource manager - at least in a JCA context - can't call setRO(), since there is no access to the actual transaction in which it is enlisted. Of course resource adapters may use hacks to get around that, but it is out of spec. So, the only way to tell the transaction manager that something is wrong is through the exception states associated with its XAResource implementation."

This is clearly wrong. A Resource Manager can always force a transaction to rollback during prepare! There's this little thing called a status which can be returned from the ... prepare method ... on the XAResource You could even throw an exception if that is preferable. Oh and you could register a Synchronization and have beforeCompletion fail - that'll cause the transaction to roll back too. So there's at least 3 different ways for you

[jesper.pedersen]

Yes, precisely - the prepare phase ("... through the exception states associated with its XAResource implementation."). My point is that a resource adapter can't call transaction.setRollbackOnly(), but a JCA container can. Side note, only JCA 1.6+ resource adapters can register a Synchronization instance.

Once ut.commit() is called, you should know all branch id's, and their state. So why doesn't it make sense to make the commit/rollback decision based on that (in addition) ?

The question is what should happen, and by whom.

In the original example 'xar' and 'xar1' in the last example are dead when commit() / rollback() is called, so anything from prepare and forward should explode (ideally 'must' with the right flags / exception). TMFAIL was sent to the EIS, so its associated entry should be in a "failed" state by that point.

Remember, ut.commit() is allowed in the original example.

[tomjenkinson]

A small clarification, Narayana won't actually allow you to re-enlist a TMFAIL XAResource in the same transaction (narayana/TransactionImple.java at master · jbosstm/narayana · GitHub)

In response to "what and whom" - the person who sets TMFAIL could also want to setRO semantics depending upon applicable specifications. Interpretation of XA and JTA specs does not suggest that when a pass-through of TMFAIL through delistResource happens, the TM would be required to also setRO hence Narayana does not do so. It may be that the JCA specification requires a setRO during the scenario you are expecting and you can then make that invocation.

[jesper.pedersen]

"

TMFAIL

The portion of work has failed. A resource manager might choose to mark a

transaction branch as rollback-only at this point. In fact, a transaction manager

does so for the global transaction. If a resource manager chooses to do so also,

xa_end() returns one of the [XA_RB∗] values. TMFAIL cannot be used in

conjunction with either TMSUSPEND or TMSUCCESS.

"

The JCA specification doesn't say anything about that the container must call setRO() once TMFAIL is sent.

IronJacamar will add a system property that controls if setRO() is called once a fatal error has occurred, and TMFAIL has been sent to the EIS. Default will be true, but frameworks / servers can choose to set it to false depending on their requirements.

[marklittle]

Jesper, you're reading one part of the specification out of context. I can do that too but it'll be as much help as your quote

So let's look at delistResource TMFAIL, which says for the flag option ...

"TMFAIL - the resource is dissociated because part of the work has failed. This typically can be caused by an error exception encountered on the resource in use."

No mention of rolling back the entire transaction there, right?

Now if you go and look at the implementation of delistResource, as Tom points out we will mark the transaction as rollback-only if certain operations fail (e.g., end) and TMFAIL is returned there. However, we must make a distinction between them returning TMFAIL and someone passing in TMFAIL to delistResource. Same error value of course, but different meanings. TMFAIL on delistResource is not a shorthand for "roll back the global transaction too please". And also recall/realise that delistResource can be called from anywhere, i.e., it doesn't have to be JCA calling it, particularly if we're running embedded outside of JCA.

[marklittle]

"It may be that the JCA specification requires a setRO during the scenario you are expecting and you can then make that invocation."

Agreed and I've got to wonder why that isn't the case here. Anything that has a handle on the instance to call delistResource can surely also then call setRollbackOnly ON THE SAME INSTANCE.

[jesper.pedersen]

And that is precisely the point - the only way is to trust the implementation of the XAResource.end() method to throw an exception if it is passed a TMFAIL flag, and it can't access the Enterprise Information System, in order to signal that the transaction should be put into MARK_FOR_ROLLBACK. The transaction manager should be able to say "hmm, something maybe fishy here that I need to do an extra verification on of during commit/rollback".

Anyway, IronJacamar will now try to delist the XAResource on the connection that had the fatal error occur on it before it is killed, and if configured, call setRO() if the transaction is currently uncommitted.

Some Enterprise Information System has really scary default behavior in error situations - like Oracle's "Oh, I lost the connection to the client, fair enough, I'll just commit all the work instead of a rollback" - so configuration options are good

[marklittle]

If you're calling delistResource with TMFAIL due to an end() throwing TMFAIL then *you* forcing the transaction to rollback is appropriate. But *you* have to do that - the transaction manager doesn't have the information to know what happened prior to the call to delistResource to realise end() failed and it can't make that a default assumption - at least not without violating XA semantics for all of those other users who call delistResource for some other reason, i.e., calling because end() didn't fail.

[jesper.pedersen]

No, I'm sending the delistResource(xar, TMFAIL) to you guys I basically don't care what the return value nor if an exception is thrown, as IronJacamar can't do nothing about that: [JBJCA-1275] Send TMFAIL and do setRollbackOnly before connection is … · ironjacamar/ironjacamar@2f29696 · GitHub

The connection will be killed shortly after, once haltCatchFire() is done.

[marklittle]

OK so why did you think we should be setting the transaction into rollback-only mode? Is this really 24 hours I'll never get back ?

[jesper.pedersen]

Well, I still think that you need to explain to the community why a single enlisted XAResource that is delisted with TMFAIL can be committed. You rely on the XAResource implementation to do the right thing in all cases, and there is no configuration option to force a certain behavior of Narayana.

At least a configuration option for a WARN if the last branch was delisted with TMFAIL would be nice... Of course the IronJacamar tracer catches these scenarios now, so there is that