7 Replies Latest reply on Feb 3, 2014 7:24 AM by cheezzario

mem leak in TransactionSynchronizer on afterCompletion() phase

cheezzario Aug 7, 2013 5:08 AM

Hello.

Let me introduce the problem

App server ( jboss-as-7.1.1.Final ) which is a part of complex distributed app using CORBA for communication fails with OutOfMemoryError

I've investigated the problem and seems like root cause is inability to clean up txSynchs ( and locks ) maps resided in org.jboss.jca.core.connectionmanager.transaction.TransactionSynchronizer class.

More detailed:

Collections mentioned above should be cleaned up in afterCompletion() method but TX is unavailable at that stage, so Transaction.hashCode() returns -1 which leads to dangled TX associations.

Right now I don't understand why javax.transaction.Transaction is used as a key in this maps? For me it seems like this code wil always fail.

I've attached modified source code of TransactionSynchronizer which I used during investigation and partial app log with records specific to one TX.

Regards, Andrey.

TransactionSynchronizer.java.zip 3.1 KB
leaked_sync.log.zip 3.9 KB

1. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

jesper.pedersen Aug 8, 2013 3:30 AM (in response to cheezzario)

Yeah, this looks fishy. I'll take a look.
Actions
2. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

cheezzario Aug 8, 2013 4:34 AM (in response to jesper.pedersen)

Seems like this problem manifests itself only when using JTS.

JBoss JTS uses com.arjuna.ats.internal.jta.transaction.jts.TransactionImple as javax.transaction.Transaction implementation class and it's hashCode implemenation uses complex delegation chain for hash code calculation (which can lead to remote calls or attempt to call an object in cleaned-up state - exactly our case: com.arjuna.ats.internal.jts.orbspecific.Controlimple is in destroyed state)

But com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple hashCode() implementation ( which is used when JTS not enabled ) simply delegates to tx.get_uid().hashCode() and TransactionSycnhronizer works as expected.
Actions
3. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

jesper.pedersen Aug 12, 2013 5:25 AM (in response to cheezzario)
Could you try and see if the follow jar solves your problem ? I'm assuming that you are using IJ 1.0.x

ironjacamar-core-impl.jar 272.0 KB
Actions
4. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

cheezzario Aug 12, 2013 10:02 AM (in response to jesper.pedersen)

Thanks a lot for your great assistance.

Good news:
Time will show but current mem dump looks OK - no signs of lost Transaction's or TransactionSynchronizer's

Bad news:
Unfortunately this jar is incompatible with jboss-as-7.1.1 ( I've upgraded jboss to 7.2.0 and verified your fix using it )
Actions
5. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

jesper.pedersen Aug 12, 2013 10:04 AM (in response to cheezzario)

Ok - thanks. I'll work on a permanent fix.

We have moved to WildFly 8 development, so use the latest Alpha from there.
Actions
6. Re: mem leak in TransactionSynchronizer on afterCompletion() phase

jesper.pedersen Aug 15, 2013 8:08 AM (in response to jesper.pedersen)

Fix has been committed. Thanks again for the report
Actions
7. Re: Re: mem leak in TransactionSynchronizer on afterCompletion() phase

cheezzario Feb 3, 2014 7:24 AM (in response to jesper.pedersen)
Hi. Seems like mem-leak problem is back in EAP 6.2

IMHO commit 9e4c08e breaks TX synchronizers map cleanup.
I've attached sample code which uses the same approach for items lookup and removal which similarly fails to remove existing item due to hashCode instability.

I've tried to use 1.0.22 in EAP 6.2 (core-impl.jar only has been replaced): leak has gone but random TX activities fail due to deadlocks in jacorb. Thread dump snippet attached.

memleak.tar.gz 56.2 KB
Actions

Go to original post