Unable to acquire lock after 15 seconds for key

[browngrey07]

The following exception is inconsistently thrown in JBoss EAP 7.0. We have not faced such an exception in JBoss EAP 6.4 or in any lower releases. What does this exception mean? Any changes is required in the web application in order to deploy the same in JBoss EAP 7.0?

ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-29) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key kJ1spSdrHHQ5kBrDLR0FCdiR5Ab0k-P4KBr_9N5X and requestor GlobalTransaction:<null>:157:local. Lock is held by GlobalTransaction:<null>:156:local

at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:190)

at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)

at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:113)

at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)

at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)

at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)

at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)

at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)

at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)

at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)

at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)

at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:72)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:62)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:235)

at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:148)

at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)

at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:792)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

[rvansa]

The lock cannot be acquired when there's too much contention on given key, or the transaction that modifies this key takes too long to finish. This operation is reading with forced lock, an equivalent of SQL's SELECT FOR UPDATE. Another your transaction is either doing the same or modifying the key in a different way, and takes too long to commit.

[browngrey07]

We have not used infinispan in our web application. This exception is raised by JBoss EAP 7.0. We are not facing this inconsistent infinispan exception in JBoss_EAP_6.4/Websphere_9/Weblogic_12c. We are facing this exception only in JBoss EAP 7.0 when we use the default standalone-full.xml as server-config file. Any config change is needed to overcome this exception?

[rvansa]

pferraro Are you aware of any change in EAP 7.0 that could lead to txs not finishing soon? Btw. is there any design doc for session replication/guidance to configuration?

[pferraro]

browngrey07 In general, please direct EAP-specific questions to Stack Overflow | Red Hat Developers

Can you describe how your web application accesses the HttpSession?  Do you ever have concurrent requests for the same session?  Does your application make use of javax.servlet.AsyncContext, either directly or via a framework?  If the answer is no to the above, I would recommend modifying your default cache configuration of your "web" cache container to use mode="SYNC" instead of "ASYNC" (these defaults have changed in EAP 7.1).

Does a typical web response include links to multiple artifacts (css, images, etc.) that require accessing the session?  If the answer is yes, this is another means of introducing session contention.

To tune your cache configuration for concurrent distributed web session access, you'll need to relax the session access constraints on the web session, which, by default, are overly strict for JEE spec compliance reasons.  EAP 7.0.4 introduces a cache configuration named "concurrent" for this purpose.  Using this configuration, a typical web request will result in multiple short-lived lock acquisitions using a last-write-wins approach, instead of a single exclusive lock acquisition for the duration of the request.  You can switch to this cache configuration by default using the following management operation:

/subsystem=infinispan/cache-container=web:set-attribute(name=default-cache, value=concurrent)

To use this configuration on a per-application basis, use the following in your jboss-web.xml:

<replication-config>
    <cache-name>web.concurrent</cache-name>
</replication-config>

If your application uses concurrent web session writes, but to different session attributes, you can further reduce cache contention by changing the way your the distributed session manager maps a session to its backing cache.  By default, all session attributes for a given session are stored in a single cache entry.  While this retains any inter-attribute object references, it will introduce contention if concurrent requests want to add, remove, or modify a session attribute.  By switching to a per-attribute cache mapping strategy where each session attribute is stored in a separate cache entry, you can not only reduce write contention when concurrent requests write to different session attributes, but you can potentially minimize the size of the replication payload generated by a given session modification (which can further reduce lock contention).  Just be aware that this will result in losing inter-attribute object referential integrity.  To use per-attribute cache mapping, use the following in your jboss-web.xml:

<replication-config>
    <replication-granularity>ATTRIBUTE</replication-granularity>
</replication-config>

You can further reduce contention for concurrent session attribute writes by optimizing the marshaling of your session attributes.  This can be achieved by traditional Java serialization customization, or more optimally by providing https://github.com/jbossas/jboss-eap7/blob/7.0.x/clustering/marshalling/api/src/main/java/org/wildfly/clustering/marshal… implementations to your web application.  Externalizers eliminate the need to serialize class descriptors along with your session attributes, and can typically reduce the serialized size by tens or even hundreds of bytes per object.  Externalizer classes are loaded via the java.util.ServiceLoader mechanism and should be enumerated within /META-INF/services/org.wildfly.clustering.marshalling.Externalizer.

[kobogianhitis]

Having applied this proposed configuration ( name="concurrent" ) on wildfly10 (full-ha) has fixed the problems with "unable to aquire lock". Indeed session is modified too often from our application, possibly in multiple places.

I'm now in the process of upgrading to wildfly12, and I wonder whether I still need to perform any changes.

In the configuration of wildfly12 (full-ha) I see there is no concurrent preset, and the default one ( name="dist" ) uses mode=SYNC. But the effective value of locking isolation is now REPEATABLE_READ, instead of READ_COMMITTED.

Should I manually switch to READ_COMMITTED (as it was in wildfly10 concurrent preset)?