JBOSS EAP 6.2 Session Lock Issue

[eopsscb]

Hi Team,

Can anyone help me in fixing the below issue. This occurs frequently in production with clustered mode. The users are facing very slow response with clustered mode and the performance is faster while running one node alone. The request taking more than 30 seconds and timed out to user with clustered mode and with single node it is responding < 5 seconds. I suspect that the issue is on cluster communication. Many portal suggested to enable sticky session, but application will not proven for sticky, we want the load to be balanced equally with round robin mode with proper session replication. We have enabled DB persistence for storing web sessions with no luck.

Configuration as below.

Mod cluster config

   <mod-cluster-config advertise-socket="modcluster" proxy-list="x.x.x.x:6666" balancer="mybalancer" advertise="false" auto-enable-contexts="true" sticky-session="false" connector="ajp">

TCP Config

  <stack name="tcpping">

                <transport type="TCP" socket-binding="jgroups-tcp"/>

                <protocol type="TCPPING">

                    <property name="initial_hosts">

                        IP1[7700],IP2[7700],IP1[8100],IP2[8100]

                    </property>

                </protocol>

DB strore config

<cache-container name="web" aliases="standard-session-cache" default-cache="database" module="org.jboss.as.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <invalidation-cache name="database" mode="ASYNC" batching="true">

                    <mixed-keyed-jdbc-store datasource="java:/jdbc/EOPSSEC" preload="true" passivation="false" purge="false">

                        <property name="databaseType">

                            oracle

                        </property>

                        <string-keyed-table prefix="JBS">

                            <id-column name="id" type="VARCHAR2(200)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </string-keyed-table>

                        <binary-keyed-table prefix="JBB">

                            <id-column name="id" type="VARCHAR2(200)"/>

                            <data-column name="datum" type="BLOB"/>

                            <timestamp-column name="version" type="NUMBER"/>

                        </binary-keyed-table>

                    </mixed-keyed-jdbc-store>

                </invalidation-cache>

            </cache-container>

11:40:13,747 ERROR [org.apache.catalina.connector] (ajp-executor-threads - 23790) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of QKkU0Dg0b2CCb1Q8CVYKgkxK

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:494) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

at org.apache.catalina.connector.Request.doGetSession(Request.java:2616) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.catalina.connector.Request.getSession(Request.java:2361) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:511) [jbossweb-7.3.2.Final-redhat-1.jar:7.3.2.Final-redhat-1]

at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)

at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:828)

at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_67]

at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010213: Cannot acquire lock default-host/eopshub/QKkU0Dg0b2CCb1Q8CVYKgkxK from cluster

at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:400)

at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:492) [jboss-as-web-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]

... 16 more

[pferraro]

You should always use sticky sessions with this particular configuration, otherwise, lock timeouts are inevitable, as subsequent requests for the same session will likely be served by a different node for each request, and thus each node will first have to obtain a distributed lock to obtain exclusive access to the session (per §7.7.2 of the Servlet specification)

Also, I assume that you have one database serving as a persistent store for all nodes?  If so, your mixed-keyed-jdbc-store must use shared="true".

[andey]

- Ensure sticky sessions are enabled.

- Most likely, your load balancer will rely on the jvmRoute to determine session affinity.

- For mod_jk, verify the worker name & node name. Worker names have to be same as node names(jvmRoutes). Please refer following to the following example:

# Set properties for extMarket11(ajp13)

worker.extMarket1.reference=worker.template

worker.extMarket1.host=x.x.x.101

worker.extMarket1.port=7109

# Set properties for extMarket12(ajp13)

worker.extMarket2.reference=worker.template

worker.extMarket2.host=x.x.x.102

worker.extMarket2.port=7109

# Set properties for extMarket21(ajp13)

#worker.extMarket3.reference=worker.template

#worker.extMarket3.host=x.x.x.103

#worker.extMarket3.port=7109

# Set properties for extMarket22(ajp13)

#worker.extMarket4.reference=worker.template

#worker.extMarket4.host=x.x.x.104

#worker.extMarket4.port=7109

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=extMarket1,extmarket2,extMarket3,extMarket4

worker.loadbalancer.method=Session

worker.loadbalancer.sticky_session=True

Node name list:

extMarket1

extMarket2

extMarket3

extMarket4

For Hardware load balancer, we still need ensure sticky mechanism are enabled

There're below Root Cause

====================

- Sticky sessions are not being maintained, allowing the session to hop across the cluster rapidly to run into these lock issues.

  - mod_proxy_balancer doesn't maintain stickiness for balancer defined outside of vhost

  - mod_cluster is not maintaining sticky sessions

- Performance issues on some cluster members are causing the load balancer to failover requests to hit the cluster lock issues.

- If worker name is different from node name, sessionID would not be complete, which cause the issue.