Using JBoss JTA embedded in a webapp: leak because of Reaper

[mauromol]

Hello all!
I hope someone can help us to find a workaround to this problem.

We're using JBoss Transactions JTA 4.4 embedded in our web application. Our webapp is deployed on a plain application server (Tomcat, for instance), so we deliver JBoss JTA libraries together with our webapp in WEB-INF/lib.

We use the JBoss JTA Transactional Driver and a JTA infrastructure coordinated by the Spring Framework and, after some months of experiments and tunings, we now have a JTA transactional webapp which works very well at driving three different DBs (among which one is managed by Hibernate).

We are now facing a problem in the webapp lifecycle, though: as soon as the first transaction is started, two new threads are created by TransactionReaper: they are ReaperThread and ReaperWorkerThread. These are daemon threads and I think they're supposed to terminate when the JVM is stopped.

However, as I said we are in a webapp environment: when our webapp is stopped, these threads remain alive, because the JVM (used by the application server) is not terminated.

Then, because of the fact that JBoss JTA classes are loaded by the webapp class loader, also ReaperThread and ReaperWorkerThread are in the context of our webapp class loader. If they stay alive even when the webapp is stopped, they prevent the webapp class loader to be garbage collected.

This causes a memory leak in webapp restarting.

Having a look at JBoss JTA sources, we found a potential solution: patching TransactionReaper by adding a "dispose" method which terminates the ReaperThread and the ReaperWorkerThread. Then, this method should be called by Spring on shutdown.

However, we are wondering if there is another workaround to get the same result without having to patch the JBoss JTA code by ourselves. Unfortunately, the impression is that JBoss Transactions has been designed with a JVM-wide lifecycle in mind: so, we can't find any method to "start" or "stop" it. Start is implicitely done when invoking static methods, while we couldn't find any code to "stop" or "reset" the library...

Any help will be appreciated.

Mauro.

[marklittle]

There are methods to start and stop the transaction manager (check the docs), but unfortunately they would not help you here because they don't do anything with the reaper. Needless to say, stopping or resetting the transaction system is something that would have to be handled very carefully, e.g., just because you think it is safe to shutdown the reaper doesn't actually mean that it is safe.

Your best bet is to wait for a feature enhancement. I've created a JIRA to cover this so feel free to monitor it.

[jhalliday]

Do you have to deploy the transaction manager with the webapp? It's designed to be part of the container, not part of the contained app.

Our own tomcat integration work deploys JBossTS as a service available to any webapp, which gets around the threading problems and some additional issues with socket allocation that can arise when multiple copies of JBossTS run concurrently, as would occur if you deploy several webapps that bundle it. I'd definitely recommend installing the transaction manager into tomcat rather than bundling it with the app, but if you can't do that then patching the code may be your only alternative in the short term.

http://anonsvn.jboss.org/repos/labs/labs/jbosstm/workspace/jhalliday/tomcat-integration/

[jhalliday]

It occurred to me whilst I munched happily on my lunch, that I'll probably have to fix JBTM-462 sooner rather than later. If I don't then undeployment of the TransactionService in JBossAS will also leak against the classloader, which will cause me to get Brocked :-(

[mauromol]

There are methods to start and stop the transaction manager (check the docs), but unfortunately they would not help you here because they don't do anything with the reaper.

Thank you Marc! Would you be so kind to please provide a hint for this?

Needless to say, stopping or resetting the transaction system is something that would have to be handled very carefully, e.g., just because you think it is safe to shutdown the reaper doesn't actually mean that it is safe.

Well, even if it would be safer to let the reaper go on after webapp stop (but I'm pretty sure it wouldn't in our case), it couldn't do much then, because the webapp has been stopped and many operations (like the creation of a new object) could fail, as the class loader is in an inconsistent state.

Your best bet is to wait for a feature enhancement. I've created <a href="https://jira.jboss.org/jira/browse/JBTM-462">a JIRA</a> to cover this so feel free to monitor it.

Thank you for this! I will monitor it.

Mauro.

[mauromol]

"jhalliday" wrote:
Do you have to deploy the transaction manager with the webapp? It's designed to be part of the container, not part of the contained app.

Our own tomcat integration work deploys JBossTS as a service available to any webapp, which gets around the threading problems and some additional issues with socket allocation that can arise when multiple copies of JBossTS run concurrently, as would occur if you deploy several webapps that bundle it. I'd definitely recommend installing the transaction manager into tomcat rather than bundling it with the app, but if you can't do that then patching the code may be your only alternative in the short term.

http://anonsvn.jboss.org/repos/labs/labs/jbosstm/workspace/jhalliday/tomcat-integration/

Well, we have worked on setting up a JBossTS configuration in order to deploy it with our webapp, so that it is used just by our webapp.

Which socket allocation issues do you think that could arise? Are you talking about the features related to the ObjectStore?

Deploying JBossTS with our own webapp has the benefit that we can setup up an environment:
- easy deployable into any application server
- that does not need any application server configuration change
- that can be setup, run and accessed in the exact same way when doing unit testing

Mauro.

[marklittle]

"jhalliday" wrote:
It occurred to me whilst I munched happily on my lunch, that I'll probably have to fix JBTM-462 sooner rather than later. If I don't then undeployment of the TransactionService in JBossAS will also leak against the classloader, which will cause me to get Brocked :-(

There used to be a capability to stop the reaper and force it to exit (some of the code still seems to be in there). I'll take a look at it tonight and maybe have a quick chat with Andrew since he was the last person to get his hands dirty with that area of the code.

[marklittle]

BTW Mauro, are you running the recovery manager?

[mauromol]

BTW Mauro, are you running the recovery manager?

As of now, we haven't needed to run it yet.

Mauro.

[marklittle]

"mauromol" wrote:

BTW Mauro, are you running the recovery manager?

As of now, we haven't needed to run it yet.

Mauro.

You do realise you need to run it and factor it in to your deployment somehow, right?

[mauromol]

There used to be a capability to stop the reaper and force it to exit (some of the code still seems to be in there). I'll take a look at it tonight and maybe have a quick chat with Andrew since he was the last person to get his hands dirty with that area of the code.

As of now, there's a boolean variable "_shutdown" that is checked by ReaperThread/ReaperWorkerThread when it is not in waiting state and that is set to true only when the shutdown() method of ReaperThread/ReaperWorkerThread is called. However, I couldn't find any other class that calls that method.
Moreover, if the threads are waiting, simply calling shutdown() wouldn't help too much: the threads have to be interrupted, too, in order to make them stop, otherwise a timeout must occur. It isn't clear to me if that timeout is related to transaction timeout or if it is another kind of timeout...

Mauro.

[mauromol]

You do realise you need to run it and factor it in to your deployment somehow, right?

Yes, I know. By now we've configured the ObjectStore to save recovery data on a disk-based DBMS, so that when (and if) we will need to do crash recovery we will start from there.

It is in our to-do list to learn how to use the recovery manager.

Mauro.

[marklittle]

It should be running continuously though and not just when someone notices that there are entries in the object store.

[mauromol]

By the way, I found two another leaks in cases like mine, this time caused by com.arjuna.ats.arjuna.coordinator.TxControl and com.arjuna.ats.arjuna.recovery.TransactionStatusManager.

In the former case, TxControl registers a shutdown hook and there's no way to remove it.

In the latter case, TransactionStatusManager creates and starts a com.arjuna.ats.internal.arjuna.recovery.Listener thread that can be interrupted just with a call to com.arjuna.ats.arjuna.recovery.TransactionStatusManager.finalize(): apart from the fact that I find it quite strange to have to call a finalization method in order to shut down something, this finalizer is actually called just by the shutdown hook registered by TxControl. It seems to me there's no way to call it from the outside, because a TransactionStatusManager instance is statically created by TxControl and hold on a static field with package visibility and no public method to access to it.

Should I open a new JIRA report for this?

Mauro.

[adinn]

There used to be a capability to stop the reaper and force it to exit (some of the code still seems to be in there). I'll take a look at it tonight and maybe have a quick chat with Andrew since he was the last person to get his hands dirty with that area of the code.

ReaperThread appears to have implemented a public shutdown() method since 4.2.1. This does not appear to have been called by the TransactionReaper in 4.2.1 or later releases. Since the thread is private this means there was no shutdown API from 4.2.1 onwards.

I added a similar shutdown method to class ReaperWorkerThread when I created it in 4.3.0. Given the lack of a shutdown API for TransactionReaper I did not add any call to this latter shutdown method.

If Transaction Reaper simply calls the two private shutdown methods for the Reaper and ReaperWorker threads then they will exit. However, this may still leave transactions running without anything to time them out which may or may not have the desired effect. I'm not sure what would be better but here are some thoughts.

Preventing the transaction from committing might be appropriate (if we do this as per the ReaperWorker thread that means calling e.cancel() and, if that fails then e.prevent_commit()).

Waiting for all currently active transactions to remove themselves from the reaper list before returning from the shutdown API might also be desirable.

Interrupting transactional threads is a non-starter as far as I am concerned. For a start what threads? There may not be anything running any code associated with the TX at the point of shutdown. Even if there is a candidate thread, interrupting it at an arbitrary point is a really, really dumb idea (no, don't ask me how dumb -- you will regret it).

So, if we a do add a public shutdown call on TransactionReaperaddingthen it might be useful to add flags to configure whether to call cancel/preventCommit and/or whether to wait for TX exits.