3 Replies Latest reply on Jul 25, 2012 6:31 PM by mircea.markus

    Serializing and retrying transactions in Infinispan

    ferwasy

      I am running a development environment of Infinispan with the following characteristics.

       

      -Two embedded instances of Infinispan running in two different instances of a web application in replicated mode, both using the following configuration:

       

      <clustering mode="replication">
      <sync />
              <stateRetrieval fetchInMemoryState="false" />
      </clustering>
      <locking  useLockStriping="false" />
      <deadlockDetection enabled="true" spinDuration="500" />
       <transaction
            transactionManagerLookupClass="..."
            syncCommitPhase="true" syncRollbackPhase="true" useEagerLocking="false"
            useSynchronization="false" eagerLockSingleNode="false"  >
                  <recovery enabled="true" />
      </transaction>
      

       

      I start a JMeter test, that makes an infinite sequence of put() operation on the SAME key to both instances concurrently. On a random pattern, I get the following exceptions:

       

      1)

       

      2012-05-29 04:36:28,166 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error 
      org.infinispan.util.concurrent.locks.DeadlockDetectedException: Deadlock found and we RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1700-4511579-61-61-301-92-249, 0000000000-1-110-62-1700-4511579-61-61-301-912 >, internalId=562962838335742} DldGlobalTransaction{coinToss=3934574857706133337, isMarkedForRollback=false, lockIntention=609575791-01, affectedKeys=[], locksAtOrigin=[609575791-01]} GlobalTransaction:<plnxdatagrid2-50485>:23392:remote shall not continue. Other tx is RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1900-1044679-61-68-6501-69-12649, 0000000000-1-110-62-1900-1044679-61-68-6501-69-122 >, internalId=562958543369060} DldGlobalTransaction{coinToss=4848346868063587289, isMarkedForRollback=false, lockIntention=null, affectedKeys=[609575791-01], locksAtOrigin=[]} GlobalTransaction:<plnxdatagrid0-16417>:24676:local 
      at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:108) 
      at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:232) 
      at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:125) 
      at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:109) 
      at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:294) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      [...]
      at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
      

       

      2)

       

      2012-05-29 04:47:53,729 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error
      org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [2001259886] for requestor [RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1700-4511579-61-61-301-467349, 0000000000-1-110-62-1700-4511579-61-61-301-4674 >, internalId=562962838335265} DldGlobalTransaction{coinToss=-4894804133801658192, isMarkedForRollback=false, lockIntention=2001259886, affectedKeys=[], locksAtOrigin=[2001259886]} GlobalTransaction:<plnxdatagrid2-50485>:25931:remote]! Lock held by [RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1800-85-10579-61-79-5801-284349, 0000000000-1-110-62-1800-85-10579-61-79-5801-2844 >, internalId=562954248401193} DldGlobalTransaction{coinToss=-8049922987111174548, isMarkedForRollback=false, lockIntention=null, affectedKeys=[2001259886], locksAtOrigin=[]} GlobalTransaction:<plnxdatagrid1-32154>:27011:local] 
      at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:240) 
      at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:133) 
      at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:80) 
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) 
      at org.infinispan.interceptors.DeadlockDetectingInterceptor.handleDataCommand(DeadlockDetectingInterceptor.java:106) 
      at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPutKeyValueCommand(DeadlockDetectingInterceptor.java:64)
      [...]
      at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
      

       

      3)

       

      2012-05-29 04:36:28,166 ERROR [org.infinispan.transaction.TransactionCoordinator] Error while processing PrepareCommand 
      3985    org.infinispan.CacheException: com.arjuna.ats.jta.exceptions.InvalidTerminationStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.invalidstate] [com.arjuna.ats.internal.jta.transaction.arjunacore.invalidstate] The transaction is in an invalid state! 
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:282) 
      at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:93) 
      at org.infinispan.transaction.xa.TransactionXaAdapter.prepare(TransactionXaAdapter.java:91) 
      at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:263) 
      at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2986) 
      at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2943) 
      at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2461) 
      at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1823) 
      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94) 
      at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:176) 
      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1396) 
      at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:138) 
      at org.springframework.transaction.jta.UserTransactionAdapter.commit(UserTransactionAdapter.java:76) 
      at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
      

       

      As long as I understand, these exceptions are expected since timeouts and deadlocks occur both in "direct" put operations and in replication operations since the test is working concurrently on the same key. So, I have the following questions:

       

      1) Suppose I have two transactions that enters in a deadlock. I want that the first transaction in chronological order proceed and the other rollback. There is another option than coin toss in order to decide which transaction rollbacks, given the fact that it uses a random value? I want that transactions be committed in chronological order.

      2) I read a comment of Galder Zamarreño (http://www.delicious.com/redirect?url=http%3A//stackoverflow.com/questions/10624975/how-to-use-jta-transactions-with-infinispan) saying that "there's no functionality to retry transactions in Infinispan". Which you would consider to be the best approach in order implement this functionality?

       

      To resume: I want to execute transactions in chronological order. If a deadlock happens, I want that the "older" transaction proceed and the "newer" transaction rollback, in order to avoid missing incremental updates.

       

      Kind regards.

      Fernando.

        • 1. Re: Serializing and retrying transactions in Infinispan
          galder.zamarreno

          What Infinispan version? In latest, 5.1.5.FINAL, there should be no issues thx to lock re-ordering.

          • 2. Re: Serializing and retrying transactions in Infinispan
            ferwasy

            Galder, I am using the version 5.1.5-SNAPSHOT that I build from https://github.com/infinispan/infinispan/tree/5.1.x, and repeating the test that I described in the first post, I have the following exception

             

            org.infinispan.interceptors.InvocationContextInterceptor 09:55:44,828 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error

            org.infinispan.util.concurrent.locks.DeadlockDetectedException: Deadlock found and we RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-1-64-8809100-75-50804142300104649, 0000000000-1-1-64-8809100-75-508041423001050 >, internalId=281479271678242} DldGlobalTransaction{coinToss=38076767288496009, lockIntention=000000019-01, affectedKeys=[], locksAtOrigin=[]} GlobalTransaction:<jboss2-19052>:580:local shall not continue. Other tx is RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-112700100-10877804142100316349, 0000000000-1-112700100-108778041421003176 >, internalId=281483566646138} DldGlobalTransaction{coinToss=2903385048076400340, lockIntention=null, affectedKeys=[], locksAtOrigin=[000000019-01]} GlobalTransaction:<fluxit-ntb-75-15102>:1780:remote

                at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:110)

                at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:207)

                at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:189)

                at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:179)

                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:209)

                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:136)

                at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242)

                at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

                at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:324)

                at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:119)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:213)

                at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:85)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:132)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:91)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

                at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140)

                at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)

                at org.infinispan.transaction.xa.TransactionXaAdapter.prepare(TransactionXaAdapter.java:106)

                at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:263)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2986)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2943)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2461)

                at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1823)

                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94)

                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:176)

                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1396)

                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:138)

                at org.springframework.transaction.jta.UserTransactionAdapter.commit(UserTransactionAdapter.java:76)

                at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)

                at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)

                at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)

                at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)

                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)

                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

                at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)

            • 3. Re: Serializing and retrying transactions in Infinispan
              mircea.markus

              To resume: I want to execute transactions in chronological order. If a deadlock happens, I want that the "older" transaction proceed and the "newer" transaction rollback, in order to avoid missing incremental updates.

              No, it's only the coin-toss that can decide the transaction rollback.

              DLD will be deprecated in a future ISPN version (at least for Optimistic tx) as Incremental locking [1] will solve the problem by serializing access to resources and allowing both transactions to succeed.

               

              [1] https://issues.jboss.org/browse/ISPN-1219