Lock is not released on remote node after transaction commit
pengyan Jul 31, 2010 7:42 AMWe have a two node cluster in replication mode, and following code are executed on the first node(NodeA).
TX.Begin
cache.getAdvancedCache.lock(TestKey);
Int value=cache.get(TestKey);
cache.put(TestKey,value+1);
Tx.commit.
Normally the change will be replicated to the other node(NodeB) in the cluster, but during a continuous load testing, occasionally(more than onces in 100,000 times), there will be a TimeoutException. The stack trace looks like following.
More tests shows this just happen when TestKey is not a String and primitive type, and lazyDeserialization is enabled. If I disable lazyDeserialization the issue is gone. Further investigation on the log files shows that during this transaction, on the remote node(NodeB), there will actaully be two locks acquired. One is for the instance of TestKey, and the other one is a MarshalledValue instance. And when this error happens, it only release the lock of the MarshalledValue, and the lock for instance of TestKey will never be released.
We are using 4.0.0final, several log files are attached. Unfornately I'm not able to create a test case for now.
line 161024 shows there's only one entry in context, and line 161012 shows the transaction has successfully locked TestKey@14c, and line 161018 shows it has locked the Marshalled value, so there must be two entries in the context. The question is why it get lost. The logging entries before line 161005 shows the normal case.
And The stack trace.
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [2 seconds] on key [test.TestKey@14c] for requestor [DeadlockDetectingGlobalTransaction{replicatingTo=[], replicationThread=null, coinToss=3121282226971076589} GlobalTransaction:<changgeng-srm-6298>:8818]! Lock held by [DeadlockDetectingGlobalTransaction{replicatingTo=[], replicationThread=null, coinToss=-5363204352586943720} GlobalTransaction:<changgeng-srm-6298>:7991]
at org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:213)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:148)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:106)
at org.infinispan.interceptors.LockingInterceptor.visitLockControlCommand(LockingInterceptor.java:146)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:171)
at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:115)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:147)
at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:82)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)
at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:99)
at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:65)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:178)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:150)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:578)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:489)
-
server.log.gz 1.3 MB