Http session replication not happening in clustered environm

[vivekoswal]

Hi all,
We are setting up a cluster of servers, the specifications are as follows:

OS: Windows 2000 Professional
JDK: jdk1.3.1_04
JBoss version: jboss-3.2.1_tomcat-4.1.24 (on 2 nodes viz server1 and server2.)
Apache: 2.0.45(installed on server1)

We have included

<distributable/>

element in the web.xml of our application
similarly we have included entries for all the entity beans and session beans in jboss.xml's of all the components.

We have added the jvmRoute attribute in the {JBOSS_HOME}\server\all\deploy\jbossweb-tomcat.sar\META-INF\jboss-service.xml file
of both the nodes

server1
<Engine jvmRoute="server1" name="MainEngine" defaultHost="localhost">

server2 <Engine jvmRoute="server2" name="MainEngine" defaultHost="localhost">

We have included the following tag in the {JBOSS_HOME}\server\all\deploy\wsc.ear\wsc.war\jboss-web.xml file of both the nodes.
<replication-config>
<replication-trigger>SET_AND_GET</replication-trigger>
<replication-type>SYNC</replication-type>
</replication-config>

APACHE mod_jk2 specific Configuration is as follows:
Following lines are added at the end of {apache_home}\conf\httpd.conf file
# Include mod_jk's specific configuration file
Include conf/mod-jk2.conf
# Load mod_jk module
LoadModule jk2_module modules/mod_jk2.so

Following are the content of {apache_home}\conf\workers2.properties file.

[shm]
info=Scoreboard. Requried for reconfiguration and status with multiprocess servers.
file=anon
size=1048576

[lb:lb]
stickySession=1
debug=1
[channel.socket:server1:8009]
port=8009
host=server1
tomcatId=server1
[channel.socket:server2:8009]
port=8009
host=server2
tomcatId=server2
[ajp13:server1:8009]
channel=channel.socket:server1:8009
group=lb
[ajp13:server2:8009]
channel=channel.socket:server2:8009
group=lb
[uri:/*]
group=lb

[status:]
info=Status worker, displays runtime information

Following are the content of {apache_home}\conf\mod-jk2.conf file

<IfModule mod_jk2.c>
JkSet config.file C:/Program Files/Apache Group/Apache2/conf/workers2.properties
</IfModule>

After making the above configurations the load balancing behaviour was achieved but the session replication does'nt seem to be happening.
We started both the nodes and observed that the cluster of 2 nodes is formed. Then we made a request which went to server1. We shut down the jboss-tomcat at server1 and made a request with the same client, we expected that server2 would take the charge (session will be replicated on server2) but it does not happen and we get a session expired message.

We would appreciate any help and suggestions on this.

Thanks and regards.
popnet.

[vivekoswal]

Putting some more info in the above post...there are times when we see the following error in the JBOSS_HOME/server/all/log/server.log file:

2004-08-03 16:55:25,328 DEBUG [org.jboss.ha.httpsession.server.ClusteredHTTPSessionService] getHttpSession called for session: cIROaQ-rCKQpZmYvGPLR+w**.ankit
2004-08-03 16:55:25,328 DEBUG [org.jboss.web.catalina.EmbeddedCatalinaService41] Loading a session out of the clustered store failed
javax.ejb.EJBException: Exception in setHttpSession: javax.ejb.FinderException: cIROaQ-rCKQpZmYvGPLR+w**.ankit does not exist
at org.jboss.ha.httpsession.server.ClusteredHTTPSessionService.getHttpSession(ClusteredHTTPSessionService.java:141)
at java.lang.reflect.Method.invoke(Native Method)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
at $Proxy211.getHttpSession(Unknown Source)
at org.jboss.web.catalina.session.ClusterManager.loadSession(ClusterManager.java:582)
at org.jboss.web.catalina.session.ClusterManager.findSession(ClusterManager.java:260)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:173)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:509)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:604)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:562)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:679)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:479)


Does this indicate something?

Regards,
Vivek

[dukehoops]

your session id string appears garbled. i'm currently having the same problem and i wonder whether that contributes to broken session replication. Seems to me, the session id is garbled because jboss (tomcat?) uses incorrect locale. I'm pretty confident that once you get the session id to appear correctly, your clustering problems will resolve. Let me know if you figure out how to fix that, i will do the same if i get there first

cheers,

-nikita

[mattvincent]

We're getting a similar Exception in setHttpSession for JBoss 3.2.5.

I don't think that the garbled session ID string has much to do with it. The session ID should be "garbled". This is from ClusteredHTTPSessionService.java

This code was written by Kevin Kelley <[e-mail]>
and adapted by Thomas Peuss <[e-mail]>
@param data The bytes you want to encode
@return the encoded String
*/
protected String encode(byte[] data)
{
char[] out = new char[((data.length + 2) / 3) * 4];
char[] alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-*".toCharArray();
//
// 3 bytes encode to 4 chars. Output is always an even
// multiple of 4 characters.
//
for (int i=0, index=0; i<data.length; i+=3, index+=4) {
boolean quad = false;
boolean trip = false;
int val = (0xFF & (int) data);
val <<= 8;
if ((i+1) < data.length) {
val |= (0xFF & (int) data[i+1]);
trip = true;
}
val <<= 8;
if ((i+2) < data.length) {
val |= (0xFF & (int) data[i+2]);
quad = true;
}
out[index+3] = alphabet[(quad? (val & 0x3F): 64)];
val >>= 6;
out[index+2] = alphabet[(trip? (val & 0x3F): 64)];
val >>= 6;
out[index+1] = alphabet[val & 0x3F];
val >>= 6;
out[index+0] = alphabet[val & 0x3F];
}
return new String(out);
}

[marinac]

Hi,

I also have this error and the session is not working. Could someone advise ? Thanks.

org.jboss.web.tomcat.tc5.TomcatDeployer Looking for session with id=07by25pCarQpjKT-hUm+sw**.worker1
org.jboss.web.tomcat.tc5.TomcatDeployer Found

org.jboss.web.tomcat.tc5.TomcatDeployer Replicating session with id 07by25pCarQpjKT-hUm+sw**.worker1

org.jboss.ha.httpsession.server.ClusteredHTTPSessionService setHttpSession called for session: 07by25pCarQpjKT-hUm+sw**.worker1

org.jboss.web.tomcat.tc5.session.ClusteredSession Are current session 07by25pCarQpjKT-hUm+sw**.worker1 attributes modified? : true

org.jboss.ha.framework.server.HAPartitionImpl dests=[cphzt20:10104 (additional data: 17 bytes)], method_call=DistributedState._set(CMPClusteredInMemoryPersistenceManager-ClusteredHTTPSession, 07by25pCarQpjKT-hUm+sw**.worker1, [B@1cfd2c6), mode=2, timeout=60000

org.jboss.ha.framework.server.HAPartitionImpl real_dests=[cphzt20:10104 (additional data: 17 bytes)]

org.jboss.ha.framework.server.HAPartitionImpl responses: [sender=cphzt20:10104 (additional data: 17 bytes), retval=null, received=true, suspected=false]

org.jboss.web.tomcat.tc5.TomcatDeployer Looking for session with id=sm0QM5I9aaSDODG0dp1XQA**.worker2

org.jboss.web.tomcat.tc5.TomcatDeployer JvmRoute id is :sm0QM5I9aaSDODG0dp1XQA**.worker1

org.jboss.ha.httpsession.server.ClusteredHTTPSessionService getHttpSession called for session: sm0QM5I9aaSDODG0dp1XQA**.worker2

org.jboss.web.tomcat.tc5.TomcatDeployer Loading a session out of the clustered store failed
javax.ejb.EJBException: Exception in setHttpSession: ; CausedByException is:
sm0QM5I9aaSDODG0dp1XQA**.worker2 does not exist
at org.jboss.ha.httpsession.server.ClusteredHTTPSessionService.getHttpSession(ClusteredHTTPSessionService.java:145)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.i

[ben.wang]

My advice is to upgrade to 3.2.7 and up. Reason being that we have a new http session replication layer that is much more maintainable. It has been out for almost 1.5 years.

-Ben

[marinac]

Hi Ben,

We upgrade to 3.2.7 as you told me. The problem is now I cannot get the sticky session work anymore and I give up of the cluster because the site is using sessions. Is there any documentation I can follow specifically on how to make sticky session works on 3.2.7 ?

Thanks.

[ben.wang]

Follow this Wiki trail should get you what you need on sticky session. Let me know if there still is a problem.

-Ben

[marinac]

Please, what wiki trial?

[marinac]

I have found it, the first message. Thank you!

[marinac]

Hi Ben,

I want to use sticky session, not session replication on jboss 3.2.7.

I was able to configure well jboss 3.2.5 but now something goes wrong.

Now the server keeps the session as far as I are on the same page, so I can refresh many times and the session id and server are the same.
I have made a test.jsp.

Then navigate into the site and check the session id and the server from time to time. I can see the request is round robin between the servers. I do not understand why this is happening when navigate and not when I am on the same page and refresh again an again.

What I have done is I have changed jboss-service.xml as I found out here: http://wiki.jboss.org/wiki/Wiki.jsp?page=ClusteringFAQ.

"Since 3.2.6, HTTP session replication is based on JBossCache and more precisely the TomcatClusteringCacheservice.
In 3.2.6, 3.2.7, 4.0.1 and 4.0.1SP1, the dependency on this service is missing in Tomcat's jboss-service.xml.

To fix it, please edit /all/jbossweb-tomcat5x.sar/META-INF/jboss-service.xml and enable the <depends optional-attribute-name="CacheName<http://wiki.jboss.org/wiki/Edit.jsp?page=CacheName>"> element, update the MBean ObjectName <http://wiki.jboss.org/wiki/Wiki.jsp?page=ObjectName> as well as the element itself.

The resulting code should look as follows:
<!--
Configuration for HTTP Session Clustering using JBossCache
-->
jboss.cache:service=TomcatClusteringCache
instead of
<!--
Configuration for HTTP Session Clustering using JBossCache
-->
<!--
<depends optional-attribute-name="CacheName">jboss.cache:service=TreeCache
-->
"

[susitha]

How did you check the load balancing? What were you using in your application (stateless session bean, Stateful etc.) Let me know in detail.
-Prabath