Asynchronicity and transaction context propagation

[chtimi2]

This question is more general than specifically about JBoss TM, but I thought the developers of JBoss TM might have an idea about it. It is about whether propagating the transaction context between different threads in asynchronous invocations is possible or not. It is hard to find specific information about this topic, some technologies say specifically that they don't implement it but they never say why.

To give some context i'm studying the integration of QoS (Quality of Service) into a component-model based framework.
The components expose used and provided services, produced and consumed events, and used and provided parameters. They live inside a light container (the component manager) that manages their lifecycle and plugs the QoS into service/event/parameter invocations by AOP.
The QoS include transactional behaviour and asynchronicity (we already have implemented Asynchronous Method Invocation and Asynchronous Method Dispatch), so i have to see if they are "orthogonal" or if they restrict each other, both in current technologies and in what is theoretically possible (and if impossible why).

My question is: is there a fundamental, intrinsic incompatibility between asynchronicity and the propagation of the transaction context?
Even in the (relatively recent) EJB 3.1 spec, it is mentioned that for the new @Asynchronous methods, the transaction context will not be propagated from the caller to the callee (but they don't say why).

But is it really an absolute impossibility?
After all during a remote transactional invocation, the transaction context is propagated to a different thread (from the client thread to the server thread) by rmi , so the transaction context doesn't have to be strictly thread-local.
Of course the service that is invoked asynchronously would/might return before committing, but that is not always unacceptable (like caches replicate their changes asynchronously after a service call), and it wouldn't prevent it from sharing the same transaction context.

So if there is an impossibility it doesn't come from sharing the transactional context between threads but from asynchronicity. What do you think?

[marklittle]

"chtimi2" wrote:
This question is more general than specifically about JBoss TM, but I thought the developers of JBoss TM might have an idea about it. It is about whether propagating the transaction context between different threads in asynchronous invocations is possible or not. It is hard to find specific information about this topic, some technologies say specifically that they don't implement it but they never say why.

Check out Checked Transactions semantics. This tends to be the reason most implementations stay clear. JBossTS supported Checked Transactions, but does allow you to configure this aspect so it's possible.

You can also check out http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.9485

[chtimi2]

Check out Checked Transactions semantics. This tends to be the reason most implementations stay clear. JBossTS supported Checked Transactions, but does allow you to configure this aspect so it's possible.

You can also check out http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.9485

Thanks for ideas, i'm going to do some reading.

Also can you explain what you mean by

JBossTS supported Checked Transactions, but does allow you to configure this aspect so it's possible.

?

[marklittle]

I meant 'JBossTS supports Checked Transactions ..."

[chtimi2]

Hello Mark, I have read the paper you cited. It does a good job at presenting the problem and a possible solution, and from what i understand JBossTS seems to use this solution.

If other people are reading this, the article can be found for free at:
http://www.cs.ncl.ac.uk/publications/inproceedings/papers/598.pdf

If you don't mind i have a few questions, first about the article (to be sure i got it right) and then about the JBossTS implementation. For the sake of clarity i will post these questions in two separate posts.

[chtimi2]

Questions and comments about the article
I quote excerpts of the article, and add my questions or comments in italic. The comments might well be wrong, by all means feel free to tell me so.

1. Introduction

Although protocols exist for enforcing thread and transaction synchronisation in local and distributed environments (commonly referred to as checked transactions [1]), they assume that communication between threads is synchronous (e.g., via remote procedure call). A thread making a synchronous call will block until the call returns, signifying that any threads created have terminated.
For example, the Object Transaction Service (OTS), specified as one of the Common Object Request Broker Architecture (CORBA) Services [2] by the Object Management Group, specifies a checked transactions protocol based upon the X/Open model.

Q1:
Doesn't this mean that OTS in intrinsically synchronous? But then isn't this synchronicity contagious up to JTS/JTA?
On the other hand other readings seem to indicate that the JTA/JTS specifications don't forbid or mandate support for the scoping of a transaction context to several threads.
The question is then, is JBTS support of this functionnality accessible through standard JTA interfaces? For my personal case, would it work with declarative transactions in Spring?

2. A processing model

This looks compatible with what we would need.

3. Checked behaviour

Checked means that when the initiating transaction has committed, the commit includes the actions of the "sub-threads".
Transactions within a single thread are always checked, multithreaded transactions require additional controls to be: completion callbacks go up the thread creation hierarchy up to the root thread, the one that initiated the transaction.
Only then is the transaction committed, even though control has already been given back to the client of the root transaction.

Q3:
Can the client have some sort of Future or does the operation have to be one-way?
Can he be notified of the transaction outcome?

and the transaction 'end' may not proceed until the thread created for the invocation upon object a returns a 'synch'.

This is coherent with the notion of a checked transaction.

The thread created by the invocation upon object a cannot terminate until the thread it created for the invocation upon object b returns a 'synch';

Q4:
But this seems stranger. Doesn't it mean that the invocation of b by a is synchronous?

4. The application domain

Asynchronous invocation is implemented as a CORBA 'one-way' call.

Q5:
Does this mean checked async transactions lony work with void one-way calls, and that the client can't get a Future?

5. Using the CORBA Object Transaction Service

Instead our proposed implementation dynamically turns asynchronous invocations into deferred synchronous,
if and only if, the invocation is occurring within a transaction.
This modification to the invocation mechanism occurs at runtime, and is completely transparent to the programmer.

Again, what does this mean for the caller of the root service that initiates the transaction?
Doesn't deferred synchronous mean you get a blocking Future reference (or something else, but that holds a return value)?
Isn't that contradictory with the other times when they say the async methods need to be one-way? I'm confused.

[chtimi2]

Questions about the JBTS implementation of this functionnality

Q'1:
Is there documentation about this topic, or examples specific to JBossTS? I couldn't find a lot in the project documentation (the JTS Programmer Guide).

Q'2:
Can i continue to use my Spring declarative transactions, with JBTS embedded as the JTA transaction manager?

Q'3:
In relation to the previous post, do async services need to be one-way to be able to participate in a multi-threaded transaction?

Q'4:
Would this work with JBossPOJOCache as the transactional resource? (i'm not using a database)

Q'5:
Any gotchas and things that aren't possible that you can think of?

[mmusgrov]

So if there is an impossibility it doesn't come from sharing the transactional context between threads but from asynchronicity. What do you think?

This is true in the case of the X/Open Distributed Transaction Processing specifications. The problem with a client starting a transaction and then making an asynchronous call to a remote server is that the client may call commit whilst the server is still trying to do transactional work. To cater for this scenario X/Open disallows transaction propagation on asynchronous requests. This is in contrast to synchronous or conversational requests in which case context propagation is allowed.

[marklittle]

"chtimi2" wrote:
Questions and comments about the article
I quote excerpts of the article, and add my questions or comments in italic. The comments might well be wrong, by all means feel free to tell me so.

1. Introduction

Although protocols exist for enforcing thread and transaction synchronisation in local and distributed environments (commonly referred to as checked transactions [1]), they assume that communication between threads is synchronous (e.g., via remote procedure call). A thread making a synchronous call will block until the call returns, signifying that any threads created have terminated.
For example, the Object Transaction Service (OTS), specified as one of the Common Object Request Broker Architecture (CORBA) Services [2] by the Object Management Group, specifies a checked transactions protocol based upon the X/Open model.

Q1:
Doesn't this mean that OTS in intrinsically synchronous? But then isn't this synchronicity contagious up to JTS/JTA?

Earlier versions of the OTS (prior to 1.4) were very heavily synchronous because that was the emphasis for CORBA. Since then we adopted various message-oriented approaches to interacting with distributed objects and the OTS evolved too. However, the notion of checked transaction semantics does not go away. What happens is that it becomes even more important to enforce it (and difficult) or understand the possibilities if that does not happen.

On the other hand other readings seem to indicate that the JTA/JTS specifications don't forbid or mandate support for the scoping of a transaction context to several threads.

No, a transaction can be associated with multiple threads at the same time. Checked transactions comes in here too.

The question is then, is JBTS support of this functionnality accessible through standard JTA interfaces? For my personal case, would it work with declarative transactions in Spring?

Some of the functionality is available, e.g., through the CheckedAction class.

As to whether it can be used through Spring declarative transactions, I really don't know.

2. A processing model

This looks compatible with what we would need.

3. Checked behaviour

Checked means that when the initiating transaction has committed, the commit includes the actions of the "sub-threads".
Transactions within a single thread are always checked, multithreaded transactions require additional controls to be: completion callbacks go up the thread creation hierarchy up to the root thread, the one that initiated the transaction.
Only then is the transaction committed, even though control has already been given back to the client of the root transaction.

Q3:
Can the client have some sort of Future or does the operation have to be one-way?
Can he be notified of the transaction outcome?

Give me an example.

and the transaction 'end' may not proceed until the thread created for the invocation upon object a returns a 'synch'.

This is coherent with the notion of a checked transaction.

The thread created by the invocation upon object a cannot terminate until the thread it created for the invocation upon object b returns a 'synch';

Q4:
But this seems stranger. Doesn't it mean that the invocation of b by a is synchronous?

No, only that the parent-child relationship must be maintained if you want checking guarantees.

4. The application domain

Asynchronous invocation is implemented as a CORBA 'one-way' call.

Q5:
Does this mean checked async transactions lony work with void one-way calls, and that the client can't get a Future?

Not at all. As long as you have notification somehow that the call has completed, then you can tie that into the checking infrastructure.

5. Using the CORBA Object Transaction Service

Instead our proposed implementation dynamically turns asynchronous invocations into deferred synchronous,
if and only if, the invocation is occurring within a transaction.
This modification to the invocation mechanism occurs at runtime, and is completely transparent to the programmer.

Again, what does this mean for the caller of the root service that initiates the transaction?

It's opaque: they don't see anything unless there is a problem, e.g., a thread doesn't complete on time and in which case the transaction is forced to roll back (better to be safe than sorry).

Doesn't deferred synchronous mean you get a blocking Future reference (or something else, but that holds a return value)?
Isn't that contradictory with the other times when they say the async methods need to be one-way? I'm confused.

[marklittle]

"chtimi2" wrote:
Questions about the JBTS implementation of this functionnality

Q'1:
Is there documentation about this topic, or examples specific to JBossTS? I couldn't find a lot in the project documentation (the JTS Programmer Guide).

The docs do talk about Checked Transactions. Also look for CheckedAction class references.

Q'2:
Can i continue to use my Spring declarative transactions, with JBTS embedded as the JTA transaction manager?

Haven't tried to, so you can let me know :-)

Q'3:
In relation to the previous post, do async services need to be one-way to be able to participate in a multi-threaded transaction?

No.

Q'4:
Would this work with JBossPOJOCache as the transactional resource? (i'm not using a database)

None of what was described assumed databases for the resource managers.

Q'5:
Any gotchas and things that aren't possible that you can think of?

Not off the top of my head.

[chtimi2]

Mmusgrov I'm not sure i understand your answer, let me answer in two parts.

"mmusgrov" wrote:
This is true in the case of the X/Open Distributed Transaction Processing specifications.

Are you saying that the solution proposed by Mark wouldn't work in the case of XA transactions (2PC)?
I have read the general paragraphs and the ones that talk about asynchronicity in http://www.opengroup.org/onlinepubs/009680699/toc.pdf,
is this the specification you're talking about? I don't see where it says this, then again i didn't understand everything.

It does say this:

3.3 Association of Threads with Transaction Branches
Several threads may participate in a single transaction branch, some more than once.

But does it imply that conversely Several threads may not participate in several transaction branches. Maybe you were thinking of something more explicit elsewhere?

"mmusgrov" wrote:
The problem with a client starting a transaction and then making an asynchronous call to a remote server is that
the client may call commit whilst the server is still trying to do transactional work.
To cater for this scenario X/Open disallows transaction propagation on asynchronous requests.
This is in contrast to synchronous or conversational requests in which case context propagation is allowed.

I don't see why 2PC makes Checked Transactions semantics not applicable:
In the solution explained in http://www.cs.ncl.ac.uk/publications/inproceedings/papers/598.pdf, the "root" transaction can commit only after
all its "daughter threads" have sent it a completion message.
If so, then wouldn't the same solution work in XA just by replacing the global commit (executed on completion of all daughter threads) by a prepare?

[mmusgrov]

I have read the general paragraphs and the ones that talk about asynchronicity in http://www.opengroup.org/onlinepubs/009680699/toc.pdf ,
is this the specification you're talking about? I don't see where it says this, then again i didn't understand everything.

I was thinking of the XATMI specification http://www.opengroup.org/pubs/catalog/c506.htm which defines how clients and servers interact in a DTP environment and the conditions under which the tx context is propagated. Section 3.5.2 says that if the client issues an asynchronous request and does not expect a reply then the request must not be issued in transaction mode.

The client must gather up all replies to his outstanding service requests (the checked semantics refered to earlier in this forum thread) prior to commit.

[chtimi2]

[e-mail] wrote:

[...]

Thanks for your answers. I still have a few more questions though :)
I have looked for references to CheckedAction in the JTS Programmer Guide, and did find a paragraph about async checked transactions.
It is at page 51 of the JTS Programmer Guide: Checked transaction behaviour, but after reading it I'm still a bit confused.


A/ What are the responsabilities of JBossTS and of the programmer, and how should i override CheckedAction?

1/At page 51 of the JTS Programmer Guide: Checked transaction behaviour

Some Transaction Service implementations will enforce checked behaviour for the transactions they support, to provide an extra level of transaction integrity.
The purpose of the checks is to ensure that all transactional requests made by the application have completed their processing before the transaction is committed.
A checked Transaction Service guarantees that commit will not succeed unless all transactional objects involved in the transaction have completed the processing of their transactional requests.

What i understand is that the Transaction Service does the checks. He knows what transactional objects are involved in the transaction.
He won't commit before all of them are done processing their transactional requests (so i suppose he blocks until they're done?).


2/Then in the The JBossTS specifics paragraph:

By default, JBossTS will issue a warning if a thread terminates a transaction when other threads are still active within it; however, it will allow the transaction termination to continue.
Other solutions to this problem are possible, e.g., blocking the threadwhich is terminating the transaction until all other threads have disassociated themselves from the transaction context.
Therefore, JBossTS provides the com.arjuna.ats.arjuna.coordinator.CheckedAction class, which allows the thread/transaction termination policy to be overridden. Each transaction has an instance of this class associated with it, and application programmers can provide their own implementations on a per transaction basis.

What i understand is that when the transaction tries to commit, JBTS is aware of threads that are still active.
But then it's up to the programmer to block the transaction-owning thread until the active threads are done.
He does so by implementing a callback (the CheckedAction implementation), that will be called by JBTS before committing the transaction.

3/The javadoc for CheckedAction also says that if you don't override it, the default just prints a warning in case threads still active.

And in the source there is this comment on the check method:

Called during transaction termination if more than one thread is associated with the transaction.
The supplied information should be sufficient for application specific implementations to do useful work (such as synchronizing on the threads).

But JBTS does know which threads are still active right? Then shouldn't the programmer just need to decide whether to commit now or block?
I don't see references to Threads in the parameters of method check, maybe they are in the Hashtable parameter?
But isn't it the TS that should be the one blocking them (or not, depending on the policy defined by the programmer's override of CheckedAction)?
Otherwise isn't it a bit contradictory (at least in spirit) with the declarative approach of Container Managed Transactions?


B/ In the same paragraph, i also don't understand what this means. What does the transaction manager need to be co-located with?

Checked transactions are only possible if using a co-located transaction
manager, i.e., the use of a separate transaction manager processes does
not allow checked transactions to be provided by the system.

[chtimi2]

I was thinking of the XATMI specification http://www.opengroup.org/pubs/catalog/c506.htm
which defines how clients and servers interact in a DTP environment and the conditions under which the tx context is propagated.

Section 3.5.2 says that if the client issues an asynchronous request and does not expect a reply
then the request must not be issued in transaction mode.

The client must gather up all replies to his outstanding service requests (the checked semantics refered to earlier in this forum thread) prior to commit.

But the client does expect a reply: the requests wouldn't be one-way (even when void).
To be more concrete by client i mean a Swing client that talks to a server (i feel that by client you mean the client of the RMs, which is our server?).
Our application is typically only one AP server, which is a client of several RMs if it can give some context. The RMs and the AP server (which is a client of the RMs) are co-localised (they live in the same VM).

I'm going to start thinking about a black-box unit test, to make sure we're talking about the same thing.

[marklittle]

Let me try to put it this way: if there are any truly asynchronous invocations within the scope of a transaction and those invocations require shipping the transaction context around with them, then you need to work with the Checked Transaction infrastructure to ensure that all work done as a result of those invocations has completed. Whether you do that through some application level correlation or the underlying system does it through some magic, is immaterial as long as it's done.