Serializing and retrying transactions in Infinispan
ferwasy Jun 18, 2012 11:14 AMI am running a development environment of Infinispan with the following characteristics.
-Two embedded instances of Infinispan running in two different instances of a web application in replicated mode, both using the following configuration:
<clustering mode="replication"> <sync /> <stateRetrieval fetchInMemoryState="false" /> </clustering> <locking useLockStriping="false" /> <deadlockDetection enabled="true" spinDuration="500" /> <transaction transactionManagerLookupClass="..." syncCommitPhase="true" syncRollbackPhase="true" useEagerLocking="false" useSynchronization="false" eagerLockSingleNode="false" > <recovery enabled="true" /> </transaction>
I start a JMeter test, that makes an infinite sequence of put() operation on the SAME key to both instances concurrently. On a random pattern, I get the following exceptions:
1)
2012-05-29 04:36:28,166 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error org.infinispan.util.concurrent.locks.DeadlockDetectedException: Deadlock found and we RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1700-4511579-61-61-301-92-249, 0000000000-1-110-62-1700-4511579-61-61-301-912 >, internalId=562962838335742} DldGlobalTransaction{coinToss=3934574857706133337, isMarkedForRollback=false, lockIntention=609575791-01, affectedKeys=[], locksAtOrigin=[609575791-01]} GlobalTransaction:<plnxdatagrid2-50485>:23392:remote shall not continue. Other tx is RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1900-1044679-61-68-6501-69-12649, 0000000000-1-110-62-1900-1044679-61-68-6501-69-122 >, internalId=562958543369060} DldGlobalTransaction{coinToss=4848346868063587289, isMarkedForRollback=false, lockIntention=null, affectedKeys=[609575791-01], locksAtOrigin=[]} GlobalTransaction:<plnxdatagrid0-16417>:24676:local at org.infinispan.util.concurrent.locks.DeadlockDetectingLockManager.lockAndRecord(DeadlockDetectingLockManager.java:108) at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:232) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:125) at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:109) at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:294) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [...] at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
2)
2012-05-29 04:47:53,729 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [2001259886] for requestor [RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1700-4511579-61-61-301-467349, 0000000000-1-110-62-1700-4511579-61-61-301-4674 >, internalId=562962838335265} DldGlobalTransaction{coinToss=-4894804133801658192, isMarkedForRollback=false, lockIntention=2001259886, affectedKeys=[], locksAtOrigin=[2001259886]} GlobalTransaction:<plnxdatagrid2-50485>:25931:remote]! Lock held by [RecoveryAwareDldGlobalTransaction{xid=< 131076, 29, 28, 0000000000-1-110-62-1800-85-10579-61-79-5801-284349, 0000000000-1-110-62-1800-85-10579-61-79-5801-2844 >, internalId=562954248401193} DldGlobalTransaction{coinToss=-8049922987111174548, isMarkedForRollback=false, lockIntention=null, affectedKeys=[2001259886], locksAtOrigin=[]} GlobalTransaction:<plnxdatagrid1-32154>:27011:local] at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:240) at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:133) at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:80) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.DeadlockDetectingInterceptor.handleDataCommand(DeadlockDetectingInterceptor.java:106) at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPutKeyValueCommand(DeadlockDetectingInterceptor.java:64) [...] at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:263)
3)
2012-05-29 04:36:28,166 ERROR [org.infinispan.transaction.TransactionCoordinator] Error while processing PrepareCommand 3985 org.infinispan.CacheException: com.arjuna.ats.jta.exceptions.InvalidTerminationStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.invalidstate] [com.arjuna.ats.internal.jta.transaction.arjunacore.invalidstate] The transaction is in an invalid state! at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:282) at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:93) at org.infinispan.transaction.xa.TransactionXaAdapter.prepare(TransactionXaAdapter.java:91) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:263) at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2986) at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2943) at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2461) at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1823) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:176) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1396) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:138) at org.springframework.transaction.jta.UserTransactionAdapter.commit(UserTransactionAdapter.java:76) at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028)
As long as I understand, these exceptions are expected since timeouts and deadlocks occur both in "direct" put operations and in replication operations since the test is working concurrently on the same key. So, I have the following questions:
1) Suppose I have two transactions that enters in a deadlock. I want that the first transaction in chronological order proceed and the other rollback. There is another option than coin toss in order to decide which transaction rollbacks, given the fact that it uses a random value? I want that transactions be committed in chronological order.
2) I read a comment of Galder Zamarreño (http://www.delicious.com/redirect?url=http%3A//stackoverflow.com/questions/10624975/how-to-use-jta-transactions-with-infinispan) saying that "there's no functionality to retry transactions in Infinispan". Which you would consider to be the best approach in order implement this functionality?
To resume: I want to execute transactions in chronological order. If a deadlock happens, I want that the "older" transaction proceed and the "newer" transaction rollback, in order to avoid missing incremental updates.
Kind regards.
Fernando.