Element stays locked after cluster node restart - Could not acquire lock

[fille]

Hello,

We are running a distributed Infinispan cluster in a non-JBoss environment. Sometimes when one of the cluster nodes is restarted a few transactions get "locked".

When trying to remove the cache entry we get following error:

org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on 4020204847648056088 on behalf of transaction RecoveryAwareGlobalTransaction{xid=< 1463898948, 36, 40, 00173-67113121-100000200389350121102-7496-35-89-1296-60-65-32-483211254189-29-2, 00173-67113121-100000200389350121102-7496-35-89-1296-60-65-32-483211254189-29-20001 >, internalId=32651148838048291} GlobalTransaction:<servername>:9649:local. Lock is being held by null

It seems that the internal element lock is somehow corrupted. There seems to be no way to delete this cache entry except restarting the complete cluster. Possibly the element gets removed after some eviction timout but this is unclear.

Some actions were already taken including:

• rollback transaction via the transaction recovery JMX console
• upgrade to infinispan 5.3 because some fixed were very related to our problem (e.g. [ISPN-3556] When LockControlCommand fails on an owner, the rollback command is not sent - JBoss Issue Tracker)
• removing the lock via Infinispan api (org.infinispan.util.concurrent.locks.LockManager)

But all without success. Any ideas on how we could proceed in solving this issue would be highly appreciated. Thanks!

[rvansa]

5.3 is still quite an old version, given that now we have 7.0.1. Could you try with current version? (you will have to update your configuration file, regrettably)

[fille]

Yes, we will definitely be upgrading to ispn 7.x somewhere next year. In the meanwhile we will be running 5.3. And there is no guarantee that version 7 will fix this problem.

[rvansa]

It obviously looks like a bug. However, nobody will try to investigate/solve that in almost 2 years old version - if it is reproducible in new versions, it could get some attention and by the time you'll be upgrading to 7, there could be 7.0.x which will have the fix included.

[fille]

In fact, we were on 5.1 and due to this problem we tried to upgrade to 6.0.1. But this version was incompatible with our JVM setup. ISPN 5.3 did however contain a lot of fixes that were backmerged from 6.x and some of these fixes were really about a similar problem. ([ISPN-3736] Backport to 5.2.x for EAP 6.3 - JBoss Issue Tracker). So the hopes were high for this 5.3 version. But I see your point that we need to upgrade if we want this issue to be fixed. Thanks!

[an1310]

The 5.2.x release has a lot of cases where you could end up with stale locks.  I found and, with the team's help, fixed them.  My submissions -- one which covers this exact case -- are in the 6.0.x and 7.0.x branches.  However, if you're stuck on 5.2.x, I've backported them to my branch: an1310/infinispan at 5.2.0-BNI · GitHub

I wouldn't use that branch as-is, as it's written for my environment.  However, this should help you along.