4 Replies Latest reply on Jan 28, 2013 7:15 AM by Radoslav Husar

    Cluster Environment Session Locks

    Manuel Pinto Newbie

      Hi,


      I need some help to resolve a problem with session locks in a cluster environment.

      We have two Liferay 6.1 over JBoss 7.1.1 in standalone-ha configuration (build 7.1.1 Final) with infinispan "web" cache-container, invalidation-cache and jdbc store. The load balancer is configured in non sticky session mode.

       

      Problem: when a node processes requests that manipulate session's data, in some cases the node locks the session and never unlock it, preventing other node from processing requests for that session. The affected node never regain the locked session and keep throwing the following exception for all subsequent requests and only recover a session when other node shutdown:

       

      Note: we also tried replication-cache and distributed-cache and all locking modes but without success.

       

      Exception:

       

      16:29:52,278 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--127.0.0.1-8080-2) An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of ruxDFIayi1vrsCejzwnVQJl3

       

      at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:534) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

      at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.13.Final.jar:]

      at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]

      at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:81) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

      Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //default-host//ruxDFIayi1vrsCejzwnVQJl3 from cluster

      at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)

      at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:560) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

      at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:532) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

      ... 11 more

       

       

      Infinispan modules:

       

      Infinispan Core 5.1.6 Final and Infinispan Cachestore JDBC 5.1.6.FINAL

       

       

      The standalone-ha.xml "web" cache-container config:

       

      <cache-container name="web" default-cache="database">

      <transport lock-timeout="60000"/>

      <invalidation-cache name="database" mode="SYNC" batching="true">

      <locking isolation="REPEATABLE_READ" />

      <mixed-keyed-jdbc-store datasource="java:jboss/datasources/LPORTAL_DS" shared="true" preload="true" passivation="false" purge="true">

      <property name="databaseType">oracle</property>

      <string-keyed-table prefix="stringbased">

      <id-column name="id" type="VARCHAR2(200)"/>

      <data-column name="datum" type="BLOB"/>

      <timestamp-column name="version" type="NUMBER"/>

      </string-keyed-table>

      <binary-keyed-table prefix="binarybased">

      <id-column name="id" type="VARCHAR2(200)"/>

      <data-column name="datum" type="BLOB"/>

      <timestamp-column name="version" type="NUMBER"/>

      </binary-keyed-table>

      </mixed-keyed-jdbc-store>

      </invalidation-cache>

      </cache-container>

       

      Thanks,

      Manuel

        • 1. Re: Cluster Environment Session Locks
          Manuel Pinto Newbie

          The correct log is:

           

          17:39:00,174 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--172.16.250.105-8080-4) An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of Cvn-K+r-cBGesIBoDrakJhrO

                  at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:528) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:496) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]

                  at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:81) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

                  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

                  at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]

          Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //default-host//Cvn-K+r-cBGesIBoDrakJhrO from cluster

                  at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

                  at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)

                  at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                  ... 12 more

          • 2. Re: Cluster Environment Session Locks
            Radoslav Husar Master

            Is there any reason why not to turn on sticky sessions? It is very recommended to use sticky sessions.

             

            Also, try with the latest version, like 7.1.3.Final.

             

            Maybe AS7-4260 could help a bit too.

            • 3. Re: Cluster Environment Session Locks
              Manuel Pinto Newbie

              Hi Radoslav,

               

              Is there any reason why not to turn on sticky sessions? It is very recommended to use sticky sessions.

               

              Customer requirement...

               

              Also, try with the latest version, like 7.1.3.Final.

               

              Where i can download this version?

               

              Maybe AS7-4260 could help a bit too.

               

              We have applied that fix but without success.

               

               

              Thanks,

              Manuel

              • 4. Re: Cluster Environment Session Locks
                Radoslav Husar Master

                Where i can download this version?

                You will have to build yourself. Checkout this branch from Git https://github.com/jbossas/jboss-as/tree/7.1.3.Final and then see building instructions.

                 

                Customer requirement...

                I recommend talking to the customer and explaining that the overhead/performance cost of such decision is significant.