2 Replies Latest reply on Dec 12, 2012 6:09 AM by rpraveenkumar

    JBAS-1836 - contentious transaction enlistment and pooling

    Adrian Brock Master

      So I finally managed to produce a stress test that reliably reproduces the
      race condition I knew exists in TxManager.

      The race condition goes something like this (two enlists on the same thread):
      Thread1: getConnection and enlist in transaction
      Thread1: first use of transaction so flags == TMNOFLAGS
      Thread1: unlock() but then a context switch before the XAResource.start()
      Thread2: getConnection from sameRM and enlist in transaction
      Thread2: second use in transaction so flags == TMJOIN
      Thread2: does XAResource.start()

      Oops, JOIN before START => XAER_PROTO (protocol error).

      To avoid this problem, I have changed TxConnectionEventListener.enlist()
      to serialize contentious enlistments.

      This is a small overhead of an extra ArrayList allocation in the non-contentious
      case (besides the transaction local locks that are required anyway).

      Comment from the code:

      
       public void enlist() throws SystemException
       {
       // This method is a bit convulted, but it has to be such because
       // there is a race condition in the transaction manager where it
       // unlocks during the enlist of the XAResource. It does this
       // to avoid distributed deadlocks and to ensure the transaction
       // timeout can fail a badly behaving resource during the enlist.
       //
       // When two threads in the same transaction are trying to enlist
       // connections they could be from the same resource manager
       // or even the same connection when tracking the connection by transaction.
       //
       // For the same connection, we only want to do the real enlist once.
       // For two connections from the same resource manager we don't
       // want the join before the initial start request.
       //
       // The solution is to build up a list of unenlisted resources
       // in the TransactionSynchronizer and then choose one of the
       // threads that is contending in the transaction to enlist them
       // in order. The actual order doesn't really matter as it is the
       // transaction manager that calculates the enlist flags and determines
       // whether the XAResource was already enlisted.
       //
       // Once there are no unenlisted resources the threads are released
       // to return the result of the enlistments.
       //
       // In practice, a thread just takes a snapshot to try to avoid one
       // thread having to do all the work. If it did not do them all
       // the next waiting thread will do the next snapshot until there
       // there is either no snapshot or no waiting threads.
       //
       // A downside to this design is a thread could have its resource enlisted by
       // an earlier thread while it enlists some later thread's resource.
       // Since they are all a part of the same transaction, this is probably
       // not a real issue.
      


        • 1. Re: JBAS-1836 - contentious transaction enlistment and pooli
          Adrian Brock Master

          I've also changed the track-connection-by-tx handling in the contentious case
          to avoid overly long locking of the transaction local.

          Instead I trap two threads allocating different connections when the connection
          should be sticky to the transaction. The second thread returns the unwanted
          connection to the pool and uses the connection obtained by the first thread.

          i.e.
          Thread1: getConnection - no current transaction sticky connection
          Thread1: get a connection from the pool, but context switch
          Thread2: getConnection - no current transaction sticky connection
          Thread2: get a connection from the pool
          Thread2: register sticky connection and return it
          Thread1: recheck for sticky connection
          Thread1: there is one now, so return the connection we just got
          Thread1: return the transaction sticky connection

          In the non-contentious case, this requires an extra lock on the tx local
          (which should be minimal overhead since there is no contention)
          and an extra check of the tx local value.

          Comment from the code (BasePool.getConnection):

           // Need a new connection for this transaction
           // This must be done outside the tx local lock, otherwise
           // the tx timeout won't work and get connection can do a lot of other work
           // with many opportunities for deadlocks.
           // Instead we do a double check after we got the transaction to see
           // whether another thread beat us to the punch.
          


          • 2. Re: JBAS-1836 - contentious transaction enlistment and pooli
            rpraveenkumar Newbie

            Hi Adrian,

             

            i am using jboss 3.2.6

             

            [i am using only one local-tx-datasource]

             

            rarely i am getting the following exception:

             

            org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: enlistResource failed; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: enlistResource failed)

             

            is the above race condition you mentioned is the reason i am getting this exception?

            [i dont think this exception is coming for me because of transaction being marked as rollback or timed out transaction, if thats the case log would have contained some info regarding that.]

             

            waiting for your reply, thank you....