new requirement for synchronization ordering (WildFly is delisting resources during Sychronization.beforeCompletion)

[smarlow]

To address a memory leak issue that occurred with some resources, WildFly is de-listing resources at (interposed) Synchronization.beforeCompletion time (this is IronJacamar doing the de-list).  However, the IronJacamar Synchronization may run before all of the JPA persistence providers Synchronization.beforeCompletion's have been called.  In the below forum link, the user reported that the first Hibernate session's work was done in the transaction, but the second Hibernate sessions work is done outside of the transaction (since its after de-list).

https://community.jboss.org/message/901613 is the WildFly forum link for this issue.

One possible workaround (see above link), would be to have a way for ordering the Synchronization.beforeCompletion/afterCompletion calls.  For this particular case, I think that we need IronJacamar Synchronizations to run last (beforeCompletion/afterCompletion within the interposed sync group).

Is there anything that could be done in the Narayana project to help?

Scott

[tomjenkinson]

Hi Scott,

We can definitely look to add something in this area.

The options I see when performing synchronization ordering are:

1. Introduce some new configuration into Narayana config that allows someone to specify ordering of synchronizations. For example:

     <entry key="JTAEnvironmentBean.syncOrders">

        com.foo.Sync1

        org.bar.Sync2

    </entry>

2. Introduce some kind of annotation support for the syncs themselves to define the order. E.g. @runBefore or @runAfter, with a class name as the parameter.

3. Specify a comparator for a tx via some TransactionManager extension.

4. Perform the ordering as suggested in https://java.net/jira/browse/JTA_SPEC-4?focusedCommentId=368214&page=com.atlassian.jira.plugin.system.issuetabpanels:com… by WildFly intercepting all sync registration and doing something like option 1 above.

5. Provide some annotations on syncs that allow them to set a priority from say 1-10. Its a bit hairy as the different impls need to have an idea of each others priority to schedule themselves but its quite flexible and there is a lot of prior art for priority. In many ways it is the way that JTA spec has already gone with a binary ordering atm for interposed and none-interposed.

Do you have a preference?

Tom

[marklittle]

We can order participants within 2PC already due to capabilities in ArjunaCore. We can't do that (easily) with Synchronizations because they're not implemented using AbstractRecords. However, there is this old JIRA that I haven't got to which would fix that ...

https://issues.jboss.org/browse/JBTM-268?jql=project%20%3D%20JBTM%20AND%20resolution%20%3D%20Unresolved%20AND%20assignee%20%3D%20marklittle%20ORDER%20BY%20priority%20DESC

The reason it hasn't been done is because it's a fairly invasive change and there hasn't been any reason to up the priority. We could do it here, but if we're only talking about two implementations that need to be ordered you can achieve the same thing by registering one of them as a 2PC participant and ensuring it is the last participant in the intentions list, with the other being a vanilla Synchronization.

[smarlow]

Not sure if its clear but we are looking for local ordering of the Synchronizations for this one particular need.  We have one interposed Synchronization class (Ironjacamar) that needs to run last during beforeCompletion.  We also would like to have the Ironjacamar Synchronization run last during afterCompletion as well (within the interposed sync group) so that there is a consistence ordering of when database connections are returned to the pool. 

[smarlow]

Option 1 sounds good to me.

[mmusgrov]

I have created https://issues.jboss.org/browse/JBTM-2259 to get this feature implemented (there is PR request linked to it). The change proposed using Option 1 is straightforward, satisfies Scotts' requirement and does not carry the risks of the extra complexity introduced by the other proposals.

My approach is to make the list of class names for specifying the partial order for synchronizations available via our JTAEnvironmentBean config bean. Scott, how do you plan to make the call to initialise this list, do we need to hardcode it in our wildfly subsystem initialisation or make it available via the wildfly management model or via some other means?

I would like to query one of your requirements:

Scott Marlow wrote:

We have one interposed Synchronization class (Ironjacamar) that needs to run last during beforeCompletion.  We also would like to have the Ironjacamar Synchronization run last during afterCompletion as well

The JTA spec requires that afterCompletion synchronizations are run in reverse order from the beforeCompletion synchronizations so this requirement does not make sense in the context of the spec.

[smarlow]

Excellent, thank you for the helping with this issue.

I'm thinking that we should initialize the list from the wildfly subsystem initialization code.

With regard to the afterCompletion synchronizations being run in reverse order to beforeCompletion.  I understand that is what we currently are doing but not sure where the JTA spec requires that.  I think that the interposed synchronization afterCompletions have to run before the non-interposed syncs but within the interposed group, we can use the order that we think is correct.  Using the reverse of registration order is fine for a default but I would like to run the IronJacamar synchronization last within afterCompletion also. 

[mmusgrov]

Scott Marlow wrote:

Using the reverse of registration order is fine for a default but I would like to run the IronJacamar synchronization last within afterCompletion also.

OK, in that case I will introduce a second option to configure whether or not we need to reverse the order of afterCompletion calls.

BTW a more general solution would be to have a config method that takes a pair of synchronization class names (x, y) with the meaning that x should be before y and I then do a topological sort to come up a set of partial orders. But that would make validation a problem since the caller could request conflicting orders - would that be introducing too much complexity.

[tomjenkinson]

Michael Musgrove wrote:

Scott Marlow wrote:

Using the reverse of registration order is fine for a default but I would like to run the IronJacamar synchronization last within afterCompletion also.

OK, in that case I will introduce a second option to configure whether or not we need to reverse the order of afterCompletion calls.

BTW a more general solution would be to have a config method that takes a pair of synchronization class names (x, y) with the meaning that x should be before y and I then do a topological sort to come up a set of partial orders. But that would make validation a problem since the caller could request conflicting orders - would that be introducing too much complexity.

Hi Mike,

I don't like the idea of running all the syncs in reverse order as it could break other parts of the system we haven't foreseen?

Perhaps we should have two lists?

BeforeCompletionList = default is <spec>, could be <spec,HHH,IJ>

AfterCompletionList = default is <spec>, could be <spec,HHH,IJ>

Tom

[tomjenkinson]

Scott, the bit of the spec where ordering is discussed is under "registerInterposedSynchronization" where "Interface TransactionSynchronizationRegistry" is described. The JavaDoc is pretty much a cut-and-paste of the spec in this area so you could take a look: http://docs.oracle.com/javaee/5/api/javax/transaction/TransactionSynchronizationRegistry.html#registerInterposedSynchron…

[mmusgrov]

Tom Jenkinson wrote:

I don't like the idea of running all the syncs in reverse order as it could break other parts of the system we haven't foreseen?

Perhaps we should have two lists?

BeforeCompletionList = default is <spec>, could be <spec,HHH,IJ>

AfterCompletionList = default is <spec>, could be <spec,HHH,IJ>

Tom

We would only be reversing the usual order for synchronizations that Scott has configured and all other synchs will run in the same order as they always have. That said, I do like the idea of having separate lists for before and after calls.

[smarlow]

Two separate lists (for before/after calls) sounds good to me.

[jesper.pedersen]

In addition to sorting based on class names I think add package names too. That way it is up to each project to maintain their own internal sorting, and it makes it easier to configure -- "*, org.hibernate, org.jboss.jca"

[smarlow]

Register a Synchronization instance with special ordering semantics. Its beforeCompletion will be called after all SessionSynchronization beforeCompletion callbacks and callbacks registered directly with the Transaction, but before the 2-phase commit process starts. Similarly, the afterCompletion callback will be called after 2-phase commit completes but before any SessionSynchronization and Transaction afterCompletion callbacks.

I always read the above as two separate groupings (interposed beforeCompletion callbacks are called after all non-interposed callbacks and interposed afterCompletion callbacks are called before non-interposed callbacks.  I thought that our ordering within the (interposed/non-interposed) groups was adding a lot of value but not a spec requirement.  Do all of the JTA implementations provide ordering within the interposed/non-interposed groups?  If yes, are they all using registration order for beforeCompletion and reverse for afterCompletion?

As we know from this case, registration order is not always going to be right, due to sync interdependencies and registrations that occur in unexpected order.

[tomjenkinson]

Hi Scott,

I think we misunderstood each other. I thought you wanted to run "interposed then session" for before completion, then "interposed then session" during after completion. I now understand you to mean you want to only reverse the order the syncs are called within each specific group.

Tom