mem leak in TransactionSynchronizer on afterCompletion() phase

[cheezzario]

Hello.

Let me introduce the problem

App server ( jboss-as-7.1.1.Final ) which is a part of complex distributed app using CORBA for communication fails with OutOfMemoryError

I've investigated the problem and seems like root cause is inability  to clean up txSynchs ( and locks ) maps resided in org.jboss.jca.core.connectionmanager.transaction.TransactionSynchronizer class.

More detailed:

Collections mentioned above should be cleaned up in afterCompletion() method but TX is unavailable at that stage, so Transaction.hashCode() returns -1 which leads to dangled TX associations.

Right now I don't understand why javax.transaction.Transaction is used as a key in this maps? For me it seems like this code wil always fail.

I've attached modified source code of  TransactionSynchronizer which I used during investigation and partial app log with records specific to one TX.

Regards, Andrey.

[jesper.pedersen]

Yeah, this looks fishy. I'll take a look.

[cheezzario]

Seems like this problem manifests itself only when using JTS.

JBoss JTS uses com.arjuna.ats.internal.jta.transaction.jts.TransactionImple as javax.transaction.Transaction implementation class and it's hashCode implemenation uses complex delegation chain for hash code calculation (which can lead to remote calls or attempt to call an object in cleaned-up state - exactly our case: com.arjuna.ats.internal.jts.orbspecific.Controlimple is in destroyed state)

But com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple hashCode() implementation ( which is used when JTS not enabled ) simply delegates to tx.get_uid().hashCode() and TransactionSycnhronizer works as expected.

[jesper.pedersen]

Could you try and see if the follow jar solves your problem ? I'm assuming that you are using IJ 1.0.x

[cheezzario]

Thanks a lot for your great assistance.

Good news:

Time will show but current mem dump looks OK - no signs of lost Transaction's or TransactionSynchronizer's

Bad news:

Unfortunately this jar is incompatible with jboss-as-7.1.1 ( I've upgraded jboss to 7.2.0 and verified your fix using it )

[jesper.pedersen]

Ok - thanks. I'll work on a permanent fix.

We have moved to WildFly 8 development, so use the latest Alpha from there.

[jesper.pedersen]

Fix has been committed. Thanks again for the report

[cheezzario]

Hi. Seems like mem-leak problem is back in EAP 6.2

IMHO commit 9e4c08e breaks TX synchronizers map cleanup.

I've attached sample code which uses the same approach for items lookup and removal which similarly fails to remove existing item due to hashCode instability.

I've tried to use 1.0.22 in EAP 6.2 (core-impl.jar only has been replaced): leak has gone but random TX activities fail due to deadlocks in jacorb. Thread dump snippet attached.