TransactionLocal: values, locks and headaches
A TransactionLocal is similar to a ThreadLocal except it is keyed on the Transaction. Beyond that the documentation is rather sparse. Unlike a ThreadLocal, TransactionLocal is part of the app server, not the standard Java libraries. It's also pluggable through the use of TransactionLocalDelegate. All this means you can't be certain of much when it comes to TransactionLocal behaviour.
With the switch from the old in-memory TM to the use of JBossTS as the default transaction manager, we also see a new TransactionLocal implementation. Needless to say, it does not work quite the same way as its predecessor. This impacts JBossAS 4.0.x if users retrofit it with the JBossTS, plus all later AS versions since they use the JBossTS implementation by default.
The JBossTS version originally (pre-TS 4.2.3.SP6) worked much the same way as the in-memory version: it used a Synchronization registered on the Transaction to store the state of ThreadLocals. This involved WeakReferences, which were suspected of causing more garbage collection load than was desirable. More seriously, there are some nasty corner cases in which it should be possible to change a TransactionLocal but it's not possible to register a Synchronization.
In order to support JTA 1.1's new TransactionSynchronizationRegistry interface, a generic Object storage mechanism was implemented on Transaction instances. Basically each Transaction behaves as a Map. So TransactionLocalDelegate was reimplemented to use this storage instead of a Synchronization.
In JBossTS, javax.transaction.Transaction is implemented by TransactionImple, which has the Map as a member variable. Up until transaction resolution (commit|rollback) a Transaction is represented by the same TransactionImple instance. After that all bets are off - getTransaction() will generally return a new TransactionImple Object instance on each call i.e. after resolution, Object identity and transaction identity are no longer the same thing. That's nice for us because it means TransactionLocal state, which is part of the TransactionImple, goes away at transaction resolution time just like it's supposed to, no WeakRefs required.
But...
The locks go away too. It's possible to lock and unlock a TransactionLocal. The lock state is stored in the same Map on the TransactionImple. This is important because in JBossTS, transactions that have reached their timeout are rolled back by a dedicated Thread.
Unlike its in-memory predecessor, the reaper in JBossTS won't try to lock a TransactionLocal before rolling back the Transaction. It's not required to. So, a transaction may roll back under the feet of a Thread which has done a lock() and therefore expects the TransactionLocal's value to stay constant.
Furthermore, to avoid race conditions you can't write a TransactionLocal on a terminated Transaction, so that Thread may attempt to update the TransactionLocal and fail. Note that unlock() will fail silent if the transaction has terminated, so calling that is not a problem.
You can't lock() on a terminated transaction either, not that you would wish to since the TransactionLocal would have no value at that point. However, that NOT the same as saying you can only lock() an active transaction. The difference is small but significant, since some processing happens during the termination protocol. Specifically, you can lock when the transaction is marked rollback, preparing or rolling back.
The short version:
1) Much of the existing code that makes use of TransactionLocal essentially assumes the behaviour of the in-memory implementation, which no longer holds. In order to support the weaker, more general contract that actually holds for pluggable transaction implementations, it may need more robust exception handling.
2) The pluggable API for transactions in JBossAS should probably be revisited and more thoroughly documented so that I can avoid writing long essays on implementation changes in the future.