Interrupting thread from transaction timeout

[kentuckyham]

Hi,
I'm using JBoss 4.2.2, trying to setup transaction timeouts on stateless session bean methods using the @TransactionTimeout annotation. I'm calling this method from another stateless bean and the transaction timeout appears to be working in that I get the following message output after the timeout interval elapses:
"18:54:14,465 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id aca0828:8f87:4761c5f5:80 invoked while multiple threads active within it.
18:54:14,466 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action aca0828:8f87:4761c5f5:80 aborting with 1 threads active!"

However, the method call itself (which is just a Thread.sleep(X) for now), doesn't exit and continues processing past the timeout. It only throws an exception upon returning from the call.

Preferred behavior would be for the annotated method to be interrupted and exit out (via exception) as soon as the timeout interval is reached. I've seen this attribute as InterruptThreads in the default transaction manager service, but I can't find an equivalent in arjuna. Is there any way to do this with arjuna JTA?

[marklittle]

Interrupting threads at arbitrary points in their execution is usually considered bad practice where transactions are concerned. JBossTS will roll back the transaction on a timeout (something the old TM didn't do), but it won't interrupt any threads that may be "active" in the transaction at that point. By default we try to maintain safety to ensure application consistency.

There is a way you can enhance this capability: you can define your own CheckedAction implementation, which is called when the terminating thread ends. You could interrupt the sleeping threads yourself. However, this does require getting into JBossTS specific code. (But then again, depending on the thread being interrupted on a timeout is pretty implementation specific). If you want to check the possibility, the CheckedAction principle is mentioned in the Core transaction Programmers Guide.

[kentuckyham]

Thanks for the fast reply, Mark.
Let me give some context around our needs and maybe you might be able to suggest a JBossTS functionality that would be more suitable to our needs.
Our application makes external calls (via web services as well Tuxedo services) to 3rd party services. Occasionally, those calls hang and never return (something on the 3rd party side).

We'd like to add container-managed transaction management that will kill our local method calling that external service when it reaches some pre-defined timeout. We'd like our method to throw an exception at that timeout. With the setup we have now, we would get an exception upon the method call returning, but if that call never gets interrupted it will never return and that doesn't help us. Is there any "best practices" way of implementing this with JBossTS?

Thanks in advance.

[marklittle]

The CheckedAction approach is probably your best bet. It allows you to set on a per-transaction basis a CheckedAction instance that will be called at termination (actually it happens just before the transaction commits or rolls back). The instance will be passed all threads that were registered as participants in the transaction. Note, that doesn't mean this list is necessarily all threads that are (or were) active in the transaction; some of the threads may never have been registered with it. Once you've got the list, you can use it to interrupt your specific threads. I'd recommend only doing this on threads that you have precise knowledge about and are blocked. Of course this assumes that those threads will place nice once they are interrupted.

If this sounds like an approach you want to investigate further, definitely check out the docs first and then let me know.

[jcreynol]

Hi Mark,

Do you have any examples of this? Can a default CheckedAction be applied through configuration or do we need to alter our application code?

We have a series of Stateless Session Beans (EJB 2) that are set up to allow users to query and report via JBoss Remoting from a custom Swing App. We've recently converted off WebLogic 8.1 which would throw a timeout error back to the client and kill off the long-running threads (usually) if the transaction timeout was reached.

Ideally we'd have a configuration option in JBoss to Interrupt the threads so the exception can be immediately thrown to the client -- any code examples or assistance is very much appreciated. When users inadvertently fire off a query that might run for more than 5 minutes, we'd like to not leave them 'hanging' waiting for the response. (looking for a solution that requires the least custom coding possible as we had counted on this behavior continuing under JBoss)

Thanks!
John

[marklittle]

Unfortunately you can't set it through configuration at the moment: you have to add this directly to your code. However, it wouldn't be too hard to add this as a config option.

[jcreynol]

Thanks for the quick reply! Can you point me to any code examples? The Programmer's Guide is unfortunately very lacking in specifics on the CheckedAction and I didn't see any code examples in the jbossts download.

Thanks again!

John

[marklittle]

I believe http://www.redhat.com/docs/manuals/jboss/jboss-eap-4.3/doc/jbossts/api/jta/com/arjuna/ats/arjuna/coordinator/CheckedAction.html should help, and that there may be some examples in the tests that are part of the source.

[jcreynol]

Thanks, Mark. I'm interested in checking out the tests and CheckedAction code (and possibly tweaking the default CheckedAction to modify its behavior). However, I'm having difficulty tracking down the exact source.

We're using jboss-eap-4.2.CR1 and the jbossjta.jar in the lib directory has the following in the META-INF\MANIFEST.MF file:
SVNTag=JBPAPP_4_2_0_GA_CP01 date=200709131706

When I look at this tag, I can't find the Arjuna code. Is this in a different SVN repository? Please advise.

http://anonsvn.jboss.org/repos/jbossas/tags/JBPAPP_4_2_0_GA_CP01

Thanks, again!!

John

[jhalliday]

JBossTS is one of the many components that the JBossAS build pulls in as a binary dependency. The source it is built from is in a separate repository

http://anonsvn.labs.jboss.com/labs/jbosstm

[marklittle]

Yeah, looking in the right repo would help ;-) There is a link off labs.jboss.com/jbosstm (http://anonsvn.labs.jboss.com/labs/jbosstm/)

[jcreynol]

So I downloaded the source and modified CheckedAction to iterate through the list of Threads and call interrupt(), as discussed, but this actually created a bigger problem. What I expected to happen was to see an exception thrown back to the Swing client (again, using EAP 4.2 CR1, w/ Remoting and Socket transport), but instead, the thread is killed off and now the client NEVER receives an error. (before, once the call completed, the client would receive an error that the transaction had timed out)

We want to kill off the transaction after X minutes AND provide the client notification... any thoughts? Shouldn't the client receive an error when the ServerThread dies?

I tried the basic thread.interrupt(), as well as wrapping casting the threads in the list to org.jboss.remoting.transport.socket.ServerThread and calling shutdown() or evict(), but all with the same result... a hung client.

Any advice is, as always, greatly appreciated.

Thanks,
John

[marklittle]

Not seeing your application, it's difficult to say. But why would the client receive an error from a server-side thread termination? Also, what kind of error did you want the client to see? That the transaction terminated (rolled back, for instance)?

[jcreynol]

Thanks for the note, Mark. Now you're on to what we're seeing as the issue. :)

Note, when we change our org.jboss.remoting.transport.Connector in our jboss-service.xml from transport=socket to transport=rmi, we actually get the behavior we'd hope to see under socket. (though lose some desired behavior that socket provides)

Under RMI, when the transaction timesout, the client is immediately notified that the transaction was timedout/rolledback. Under Socket, the thread that is wrapped in the transaction actually finishes before anything is sent back to the client -- even after the transaction timed out and rolled back! So the user gets an IllegalStateException returned after waiting for the entire thread to complete.

Not the best design, but we have always counted on the transaction timeout/rollback error to notify the client immediately. Imagine a user fires off a query to load a list of tasks and they accidentally enter in a date range much larger than they intended. (granted, we could control some more at the client level, but ideally we would make this conversion to jboss w/o having to change too much of this long-standing code) Under RMI, we can send a message back to the user after 5 or 10 minutes, so they can see that the transaction timed out and clean up the query or otherwise try something else.

Under Socket, the behavior is far less desirable. Say the query runs for 12 minutes with a transaction timeout of 5 minutes. The transaction dies after 5 minutes but the end user on the client side doesn't receive a notification and instead is left to watch the status bar bounce for another 7 minutes... and at that time they don't even receive their result set, they get an IllegalStateException! (we could adjust our timeout, but that's not what the users are used to and we'd ultimately like to keep some consistency after the migration from weblogic to jboss)

What can be done to get Socket transport behaving like RMI transport? (i.e. sending a transaction timeout/rollback exception back to the client)

Thanks again for your consideration, here!

-John