So I finally managed to produce a stress test that reliably reproduces the
race condition I knew exists in TxManager.

The race condition goes something like this (two enlists on the same thread):
Thread1: getConnection and enlist in transaction
Thread1: first use of transaction so flags == TMNOFLAGS
Thread1: unlock() but then a context switch before the XAResource.start()
Thread2: getConnection from sameRM and enlist in transaction
Thread2: second use in transaction so flags == TMJOIN
Thread2: does XAResource.start()

Oops, JOIN before START => XAER_PROTO (protocol error).

To avoid this problem, I have changed TxConnectionEventListener.enlist()
to serialize contentious enlistments.

This is a small overhead of an extra ArrayList allocation in the non-contentious
case (besides the transaction local locks that are required anyway).

Comment from the code:

 public void enlist() throws SystemException
 {
 // This method is a bit convulted, but it has to be such because
 // there is a race condition in the transaction manager where it
 // unlocks during the enlist of the XAResource. It does this
 // to avoid distributed deadlocks and to ensure the transaction
 // timeout can fail a badly behaving resource during the enlist.
 //
 // When two threads in the same transaction are trying to enlist
 // connections they could be from the same resource manager
 // or even the same connection when tracking the connection by transaction.
 //
 // For the same connection, we only want to do the real enlist once.
 // For two connections from the same resource manager we don't
 // want the join before the initial start request.
 //
 // The solution is to build up a list of unenlisted resources
 // in the TransactionSynchronizer and then choose one of the
 // threads that is contending in the transaction to enlist them
 // in order. The actual order doesn't really matter as it is the
 // transaction manager that calculates the enlist flags and determines
 // whether the XAResource was already enlisted.
 //
 // Once there are no unenlisted resources the threads are released
 // to return the result of the enlistments.
 //
 // In practice, a thread just takes a snapshot to try to avoid one
 // thread having to do all the work. If it did not do them all
 // the next waiting thread will do the next snapshot until there
 // there is either no snapshot or no waiting threads.
 //
 // A downside to this design is a thread could have its resource enlisted by
 // an earlier thread while it enlists some later thread's resource.
 // Since they are all a part of the same transaction, this is probably
 // not a real issue.