Session replication problem
Hi,
There is a problem under JBoss 7.1.1 Final (not EAP) on my system.
Assume that there are two jboss standalone server;N1, N2.
N1 and N2 is clustered with JGroups and Infinispan, and the session clustering was setted without mod_jk.
This was not new bug, but I want to check about this problem with my configuration.
(AS-4260 was reported last month, I found)
Steps for Reproducing Exception
#1, Check N1, N2 is running.
#2, Login to my application through its web page, the login token is writted to session correctly.
(Automatic replication to each other with this new session value-login token)
#3, Restart N1.
#4, As soon as possible, restart N2 after N1's startup is completed.
#5, Click my application's web page related to session handling.
#6, Then, I can see below Exception message
Exception stacktrace
06:54:30,723 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--10.1.2.60-8080-1) An exception or error occurred in the container during the request processing: java.lang.RuntimeExcepon acquiring ownership of CZiYxVH+DoM6Ac79QLlIOTUP
at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:528) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:496) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:81) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]
Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //OOOO//CZiYxVH+DoM6Ac79QLlIOTUP from cluster
at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)
at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)
at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
... 12 more.
Exception is printed only 1 time, and the user login token is valid aftre showing this stacktrace, no problem. But I want to erase thie exception message.
Cause - Is this right??
I think that Infinispan's cache rebalancing is taken place between step 3 and step 4.
As the rebalancing task is blocking processing, the session lookup was blocked during some seconds. Then this blocking arise the exception described in configuration file.
Is this right?
Prevention of this Exception.
N1 restart -> sleep 5(me) -> N2 restart, not urgently. (Give enough time to JBoss for rebalancing)
This can be helpful??
How about my settings?
This is my JBoss configuration related to clustering. This is enough? My environment was in AWS EC2.
<standalone.xml>
...
<subsystem xmlns="urn:jboss:domain:infinispan:1.2" default-cache-container="web">
<cache-container name="web" aliases="standard-session-cache" default-cache="repl">
<transport stack="tcp" lock-timeout="60000"/>
<replicated-cache name="repl" mode="ASYNC" start="EAGER" batching="true">
<locking isolation="REPEATABLE_READ"/>
<eviction strategy="LRU" max-entries="200"/>
<file-store/>
</replicated-cache>
</cache-container>
</subsystem>
...
<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
<protocol type="FILE_PING">
<property name="location">
/shared_disk/cluster/jboss
</property>
<property name="timeout">
5000
</property>
<property name="num_initial_members">
2
</property>
</protocol>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
</subsystem>
<Startup script>
JAVA_OPTS = "$JAVA_OPTS -Djboss.host.name=$HOSTNAME:8080" (+offset)
JAVA_OPTS = "$JAVA_OPTS -Djboss.node.name=$APP_NAME"
. bin/standalone.sh -b $HOSTNAME --server-config="standalone.xml"
<$HOSTNAME>
The hostname is described correctly /etc/sysconfig/network, like OO.OO.OO.OO.
Can you help me?