2 Replies Latest reply on Jun 19, 2017 5:19 AM by andey

    JBOSS EAP 6.2 Session Lock Issue

    eopsscb
    eopsscb Jun 16, 2017 6:48 AM

      Hi Team,

       

      Can anyone help me in fixing the below issue. This occurs frequently in production with clustered mode. The users are facing very slow response with clustered mode and the performance is faster while running one node alone. The request taking more than 30 seconds and timed out to user with clustered mode and with single node it is responding < 5 seconds. I suspect that the issue is on cluster communication. Many portal suggested to enable sticky session, but application will not proven for sticky, we want the load to be balanced equally with round robin mode with proper session replication. We have enabled DB persistence for storing web sessions with no luck.

       

      Configuration as below.

       

      Mod cluster config

         <mod-cluster-config advertise-socket="modcluster" proxy-list="x.x.x.x:6666" balancer="mybalancer" advertise="false" auto-enable-contexts="true" sticky-session="false" connector="ajp">

       

      TCP Config

        <stack name="tcpping">

                      <transport type="TCP" socket-binding="jgroups-tcp"/>

                      <protocol type="TCPPING">

                          <property name="initial_hosts">

                              IP1[7700],IP2[7700],IP1[8100],IP2[8100]

                          </property>

                      </protocol>

       

      DB strore config

       

      <cache-container name="web" aliases="standard-session-cache" default-cache="database" module="org.jboss.as.clustering.web.infinispan">

                      <transport lock-timeout="60000"/>

                      <invalidation-cache name="database" mode="ASYNC" batching="true">

                          <mixed-keyed-jdbc-store datasource="java:/jdbc/EOPSSEC" preload="true" passivation="false" purge="false">

                              <property name="databaseType">

                                  oracle

                              </property>

                              <string-keyed-table prefix="JBS">

                                  <id-column name="id" type="VARCHAR2(200)"/>

                                  <data-column name="datum" type="BLOB"/>

                                  <timestamp-column name="version" type="NUMBER"/>

                              </string-keyed-table>

                              <binary-keyed-table prefix="JBB">

                                  <id-column name="id" type="VARCHAR2(200)"/>

                                  <data-column name="datum" type="BLOB"/>

                                  <timestamp-column name="version" type="NUMBER"/>

                              </binary-keyed-table>

                          </mixed-keyed-jdbc-store>

                      </invalidation-cache>

                  </cache-container>

       

       

      11:40:13,747 ERROR [org.apache.catalina.connector] (ajp-executor-threads - 23790) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of QKkU0Dg0b2CCb1Q8CVYKgkxK

      at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:494) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

      at org.apache.catalina.connector.Request.doGetSession(Request.java:2616) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.catalina.connector.Request.getSession(Request.java:2361) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:511) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

      at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

      at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)

      at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

      at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:828)

      at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_67]

      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

      Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010213: Cannot acquire lock default-host/eopshub/QKkU0Dg0b2CCb1Q8CVYKgkxK from cluster

      at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:400)

      at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

      at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:492) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

      ... 16 more

      • 929 Views
      • Tags:
        • 1. Re: JBOSS EAP 6.2 Session Lock Issue
          pferraro
          pferraro Jun 16, 2017 8:03 AM (in response to eopsscb)

          You should always use sticky sessions with this particular configuration, otherwise, lock timeouts are inevitable, as subsequent requests for the same session will likely be served by a different node for each request, and thus each node will first have to obtain a distributed lock to obtain exclusive access to the session (per §7.7.2 of the Servlet specification)

          Also, I assume that you have one database serving as a persistent store for all nodes?  If so, your mixed-keyed-jdbc-store must use shared="true".

            • Actions
          • 2. Re: JBOSS EAP 6.2 Session Lock Issue
            andey
            andey Jun 19, 2017 5:19 AM (in response to eopsscb)

            - Ensure sticky sessions are enabled.

            - Most likely, your load balancer will rely on the jvmRoute to determine session affinity.

             

            - For mod_jk, verify the worker name & node name. Worker names have to be same as node names(jvmRoutes). Please refer following to the following example:

             

            # Set properties for extMarket11(ajp13)

            worker.extMarket1.reference=worker.template

            worker.extMarket1.host=x.x.x.101

            worker.extMarket1.port=7109

             

             

            # Set properties for extMarket12(ajp13)

            worker.extMarket2.reference=worker.template

            worker.extMarket2.host=x.x.x.102

            worker.extMarket2.port=7109

             

             

            # Set properties for extMarket21(ajp13)

            #worker.extMarket3.reference=worker.template

            #worker.extMarket3.host=x.x.x.103

            #worker.extMarket3.port=7109

             

             

            # Set properties for extMarket22(ajp13)

            #worker.extMarket4.reference=worker.template

            #worker.extMarket4.host=x.x.x.104

            #worker.extMarket4.port=7109

             

            worker.loadbalancer.type=lb

            worker.loadbalancer.balance_workers=extMarket1,extmarket2,extMarket3,extMarket4

            worker.loadbalancer.method=Session

            worker.loadbalancer.sticky_session=True

             

            Node name list:

             

            extMarket1

            extMarket2

            extMarket3

            extMarket4

             

            For Hardware load balancer, we still need ensure sticky mechanism are enabled

             

            There're below Root Cause

            ====================

             

            - Sticky sessions are not being maintained, allowing the session to hop across the cluster rapidly to run into these lock issues.

              - mod_proxy_balancer doesn't maintain stickiness for balancer defined outside of vhost

              - mod_cluster is not maintaining sticky sessions

            - Performance issues on some cluster members are causing the load balancer to failover requests to hit the cluster lock issues.

            - If worker name is different from node name, sessionID would not be complete, which cause the issue.

              • Actions