5 Replies Latest reply on Sep 26, 2016 11:25 AM by Paul Ferraro

    Wildfly web replication (infinispan) conflicting with websocket

    Yannick Menager Newbie

      After deploying my app on an HA deployment, I'm running into infinispan locking issues that I strongly suspect are related to our use of websockets/Sockjs (log below)

       

      We are using websockets, however that part code doesn't touch the session in any way. One of the error in the logs does seem to indicate the session is being accessed during the sockjs/websocket negotiation, but it's all happening in wildfly rather than my code, specifically:

       

      io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69) which is calling getSession().

       

      Does anyone has any idea on how I can stop any session access happening when a websocket connection occurs ?

       

      2016-07-30 16:36:02,827 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-1) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:48:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)

          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:02,834 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to //ws/browser/693/5ornmcwr/xhr_send: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:48:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)

          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:17,848 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-1) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:50:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)

          at com.sun.faces.context.ExternalContextImpl.getSession(ExternalContextImpl.java:167)

          at javax.faces.context.ExternalContextWrapper.getSession(ExternalContextWrapper.java:438)

          at javax.faces.application.ViewHandler.calculateCharacterEncoding(ViewHandler.java:336)

          at javax.faces.application.ViewHandler.initView(ViewHandler.java:645)

          at com.sun.faces.application.view.MultiViewHandler.initView(MultiViewHandler.java:116)

          at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:122)

          at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)

          at javax.faces.webapp.FacesServlet.service(FacesServlet.java:658)

          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)

          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:265)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:200)

          at io.undertow.servlet.spec.RequestDispatcherImpl.error(RequestDispatcherImpl.java:453)

          at io.undertow.servlet.spec.RequestDispatcherImpl.error(RequestDispatcherImpl.java:386)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:311)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:17,850 ERROR [io.undertow.request] (default task-1) UT005022: Exception generating error page /public/error.xhtml: java.lang.RuntimeException: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:50:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at io.undertow.servlet.spec.RequestDispatcherImpl.error(RequestDispatcherImpl.java:459)

          at io.undertow.servlet.spec.RequestDispatcherImpl.error(RequestDispatcherImpl.java:386)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:311)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

      Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:50:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)

          at com.sun.faces.context.ExternalContextImpl.getSession(ExternalContextImpl.java:167)

          at javax.faces.context.ExternalContextWrapper.getSession(ExternalContextWrapper.java:438)

          at javax.faces.application.ViewHandler.calculateCharacterEncoding(ViewHandler.java:336)

          at javax.faces.application.ViewHandler.initView(ViewHandler.java:645)

          at com.sun.faces.application.view.MultiViewHandler.initView(MultiViewHandler.java:116)

          at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:122)

          at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)

          at javax.faces.webapp.FacesServlet.service(FacesServlet.java:658)

          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

          at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)

          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:265)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:200)

          at io.undertow.servlet.spec.RequestDispatcherImpl.error(RequestDispatcherImpl.java:453)

          ... 10 more

       

      2016-07-30 16:36:32,853 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-1) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:52:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

          at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:760)

          at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:32,855 ERROR [io.undertow.request] (default task-1) UT005071: Undertow request failed HttpServerExchange{ POST /public/error.xhtml request {Accept=[*/*], Accept-Language=[en-US,en;q=0.8], Accept-Encoding=[gzip, deflate], Origin=[http://10.4.0.6], User-Agent=[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36], X-Forwarded-Port=[80], X-Forwarded-For=[10.4.0.1], Content-Length=[56], Content-type=[text/plain], Cookie=[JSESSIONID=NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO.web2], Referer=[http://10.4.0.6/public/operation.xhtml?opId=AEAAU2LEOZVWK6JOMNXW2WVIOVDFP6KHG63T6SSMAWBLLMI&url=http%3A%2F%2F10.4.0.6%2Fpublic%2Flogin.xhtml], Host=[10.4.0.6]} response {}}: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web2>:52:local. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828)

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:810)

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:84)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)

          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)

          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

          at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:760)

          at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557)

          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)

          at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

          at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

          at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

          at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

          at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:47,880 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p14-t2) ISPN000136: Error executing command LockControlCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web1>:47:remote. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

       

      2016-07-30 16:36:47,882 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p14-t2) ISPN000071: Caught exception when handling command LockControlCommand{cache=idvkey-portal.war, keys=[NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO], flags=null, unlock=false, gtx=GlobalTransaction:<web1>:47:remote}: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key NKNorsp7HM4Ybtc0ZaZDq3iQpEe171K2E9LcLlPO and requestor GlobalTransaction:<web1>:47:remote. Lock is held by GlobalTransaction:<web1>:45:remote

          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)

          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:199)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)

          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

          at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)

          at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)

          at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

          at java.lang.Thread.run(Thread.java:745)

        • 1. Re: Wildfly web replication (infinispan) conflicting with websocket
          Paul Ferraro Master

          The session access in the above stack trace is due to a security-constraint being applied to your web socket request (authentication tokens are stored in the HttpSession).  Remove these constraints, and the session shouldn't be accessed.

          • 2. Re: Wildfly web replication (infinispan) conflicting with websocket
            Yannick Menager Newbie

            The problem is that I haven't added any security constraints at all in my app (security is handled by spring security not jee).

            • 3. Re: Wildfly web replication (infinispan) conflicting with websocket
              Paul Ferraro Master

              I'm not sure why you would use spring for security since JEE already provides standard mechanisms for configuring things like security constraints.  This would be a question for the spring forums.

              • 4. Re: Wildfly web replication (infinispan) conflicting with websocket
                Yannick Menager Newbie

                It's historical (the app was originally running on tomcat).

                 

                But anyway spring has nothing to do with this problem.

                 

                The problem is CachedAuthenticatedSessionHandler seems to be automatically added / called (see fragment from stack trace below) although I don't have any JEE security configured, and in fact i even removed the security provider in the configuration.

                 

                Unless there is some way to stop wildfly from adding/running CachedAuthenticatedSessionHandler, the consequence is that it's impossible to use session replication in an app that uses websockets.

                 

                    at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)

                    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)

                    at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

                    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                • 5. Re: Wildfly web replication (infinispan) conflicting with websocket
                  Paul Ferraro Master

                  You seem to have requests bouncing between nodes.  You ought to have your load balancer configured with sticky sessions to prevent this.  Once that is fixed, you can work around the locking issues by leveraging the "concurrent" Infinispan web session cache configuration.  To use this specify the following in your jboss-web.xml:

                  <replication-config><cache-name>web.concurrent</cache-name></replication-config>