1. Should this be the default? i.e. Does it affect most actors? We are only aware of one circumstance so far. We therefore propose it not be made the default at this time

I agree that setting the transaction to roll-back only should not be the default.  My preference would be to let users configure their choice and default to calling rollback directly from the reaper thread, as is done today.  My reason is to minimize the operational impact on users.  Let them opt in to the set roll-back only option if that better fits their needs.

2. Should it only trigger when Narayana detects actors of a configurable type are enlisted in the transaction

Hmm, this is an interesting idea.  If one of the actors is a JPA persistence provider Synchronization or an EE (JPA) container level Synchronization, we would want to use the configured choice (mark roll-back only or rollback), but if non of these Synchronizations are registered in the active transaction, we could use  a different configured option (mark roll-back only or rollback). 

3. How should we handle the distributed transaction scenario where the remote server crashes and a transaction has been propagated to another server. In this case the remote server would be marked rollback only but as the application thread never returns to it it won't be given the opportunity to rollback. Potentially interposed coordinators could be forced to use rollback semantics from their reapers?

An example would help me here.  Are you saying that JVM1 starts TX1 and invokes JVM2 with TX1.  In the middle of the call to JVM2, JVM2 terminates and therefore, does not return control to JVM1.  Meanwhile, in JVM1 the remote client invocation detects a socket error and throws an exception, which is caught by the EJB container that will end the transaction by rolling it back.  Alternatively, if JVM1 did not detect the socket error and is still waiting for a response from JVM2, the JVM1 reaper will eventually mark the tx as rollback-only but the socket read will continue to block until it times out.  Is this close to what happened?

I don't yet see in this case, how the (subordinate coordinator/resource???) gets the chance to use rollback semantics from their reaper (I assume you mean JVM1 in this case).  How does the subordinate coordinator detect that JVM2 crashed (is there a watch dog timer pulse in play here?) 

How about some type of liveness check where we later detect that the set rollback only didn't have any impact (on the transaction) after N seconds?

For subordinate transactions (more generally transactions where the control is not within the JVM in question), the reaper needs to know that it should still call rollback when the time-out goes off, or after some period.

Two ways of approaching this:

a) have the reaper call rollback eventually irrespective of the type of transaction (call setRollbackOnly on the timeout, then set another timeout for rollback).

b) have the reaper call setRollbackOnly on timeout and later use bottom-up recovery on inflight transactions to check the status of the parent transaction. Then act accordingly.

I don't yet understand why the distributed bug that you mentioned above (with "set rollback only"), is reason enough to dismiss the implementation idea of using SRO.  Having said that, if there are other implementations that address the issue in a better way (making the distributed case easier to solve), lets list them (and the underlying solution that they target).

One that comes to mind, would be for WildFly users to only use persistence providers that synchronize access to the underlying persistence context.  We don't have that option now and even if we did (e.g. some future version of Hibernate), we would still want to support applications that package the older version of the persistence provider.  So that doesn't work.

One underlying requirement is that the transaction manager reaper thread, be flexible about meeting the user expectation for the following goals that will vary:

1. Some users need the reaper thread to roll back JTA transactions in the background, even though this can violate the JPA concurrency requirements when the persistence provider Synchronization.afterCompletion(int status) callback mutates the underlying entity manager instance, while the application thread may be actively mutating it also.
2. Some users need the reaper thread to roll back the JTA transaction but not in a way that leads to concurrency violations of the JPA persistence context (underlying entity manager).

We already have #1 solved today, but need a way to configure for #2.

Do we have other ideas besides SRO?  I'm especially interested in ideas that target implementation in Narayana.

Why not make the reaper timeout configurable so that JPA can trigger what it needs to do after the tx timeout has passed but before the reaper timeout is breached.

Closing the EM, is a specification requirement for the EE (JPA) container.  Detaching any loaded entities, is a spec requirement for the JPA persistence provider (only if transaction was rolled back).  Detaching entities from a non-application thread will violate EntityManager concurrency.  In your proposal, which thread calls the TxAssociationListener callbacks (reaper/TM communication threads or application thread that previously owned the transaction that rolled back?

How does that prevent the reaper thread from calling the Hibernate ORM Synchronization.afterCompletion(int) that updates the Hibernate session while the application thread may be using the Hibernate session still?

Scott Marlow wrote:

 

How does that prevent the reaper thread from calling the Hibernate ORM Synchronization.afterCompletion(int) that updates the Hibernate session while the application thread may be using the Hibernate session still?

I don't have an in depth knowledge of the JPA spec so perhaps my solution is naive but I was thinking that you don't necessarily need to register any synchronizations. When the tx timeout is reached the reaper marks it rollback-only (and when the reaper timeout is reached it would roll back the transaction). This gives the hibernate layer the opportunity to detect that the transaction timeout period is passed (by setting a timer) before the reaper actually rolls back the transaction and you can do what you need to do to work around the issue.

Yes, that would also work.  Your idea sounds similar to solution a above.

The Hibernate synchronization would run in the application thread, which avoids concurrent updates to the Hibernate session.  In the distributed/multi-vm case, the abandoned jvm that doesn't know that the transaction should end, would use the second timeout to eventually roll back the transaction.