Can't get sticky sessions to work

[koreth]

I have a cluster of two JBoss 4.0.1RC1 servers sitting behind an Apache 2 server running mod_jk2. The Apache server is doing load balancing and failover, but so far I have been completely unable to get session stickiness to work. The JSESSIONID cookie never includes the jvmRoute value.

I've modified my jbossweb-tomcat50.sar/server.xml file's Engine tag:

host 1:

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="sgrimm1">

host 2:

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="sgrimm2">

My application's web.xml has the "distributable" tag.

The two hosts see each other in the cluster; I get log messages at startup like

14:37:20,056 INFO [DefaultPartition] New cluster view for partition DefaultPartition: 9 ([10.0.0.1:1099, 10.0.0.2:1099] delta: 0)
14:37:20,067 INFO [DefaultPartition] I am (null) received membershipChanged event:
14:37:20,071 INFO [DefaultPartition] Dead members: 0 ([])
14:37:20,076 INFO [DefaultPartition] New Members : 0 ([])
14:37:20,080 INFO [DefaultPartition] All Members : 2 ([10.0.0.1:1099, 10.0.0.2:1099])

I tried running a network sniffer on the traffic between Apache and JBoss, and I see that JBoss is sending back a JSESSIONID cookie that doesn't include the jvmRoute value. So I assume it's not a problem with the configuration on the Apache side. Or is mod_jk2 supposed to be the thing that appends the jvmRoute to the session cookie? Here's my workers2.properties just in case it's relevant.

[shm]
file=/tmp/mod_jk2_shm.file
size=1048576

[channel.socket:sgrimm1:8009]
[channel.socket:sgrimm2:8009]

[ajp13:sgrimm1:8009]
channel=channel.socket:sgrimm1:8009
tomcatId=sgrimm1

[ajp13:sgrimm2:8009]
channel=channel.socket:sgrimm2:8009
tomcatId=sgrimm2

[status:status]

[lb:lb]
stickySession=1

[uri:/jsp/*]
worker=lb

[uri:/jmx-console/*]
worker=lb

I originally had "[lb]" instead of "[lb:lb]" but it made no difference.

In attempting to diagnose the problem, I tried adding a debug logger to the Tomcat server.xml, using the existing logger as a guide:

<Logger className="org.jboss.web.tomcat.Log4jLogger"
 verbosityLevel="DEBUG"
 category="org.jboss.web"/>

I don't get any debug output, but I do get this message:

13:35:30,262 INFO [STDOUT] Can't find resource org.jboss.web.tomcat.session.LocalStrings org.jboss.web.tomcat.tc5.WebCtxLoader$ENCLoader@f3387
13:35:30,265 INFO [STDOUT] [Ljava.net.URL;@1e16028

(It looks like there is a LocalStrings.properties in org.jboss.web.tomcat.tc5.session, but not in org.jboss.web.tomcat.session.)

Any thoughts on how I can diagnose this, or better yet, fix it?

[ganton516]

koreth:

I JUST got all this working, the one big difference I can see between your setup and mine, is that when running the tomcats on separate machines, you do NOT need the jvmRoute, and tomcatID attributes, each jvm only has it's own route.

Here is my workers2.properties file, adjust yours accordingly.

I will put a watch on this thread, if this works, or doesn't, please let me know, I know how frustrating this is.

[logger]
level=DEBUG

[config:]
file=/data/apache/conf/workers2.properties
debug=0
debugEnv=0

[shm]
info=Scoreboard. Required for reconfiguration and status with multiprocess servers
file=/var/log/apache/mod_jk2.shm
size=1000000
debug=0
disabled=0

[workerEnv:]
info=Global server options
timing=1
debug=0

[lb:lb]
info=Default load balancer.
debug=1

# appserv1
[channel.socket:10.0.0.1:8009]
port=8009
host=10.0.0.1
lb_factor=100

# appserv2
[channel.socket:10.0.0.2:8009]
port=8009
host=10.0.0.2
lb_factor=100

[uri:/wayneh/*]
info=Waynes Website

[koreth]

Thanks for the suggestion. I modified my workers2.properties to look like yours, but still no stickiness. Apache still happily talks to both of the app servers, but if I open a browser, go to one of my JSPs, and repeatedly reload the page, I see the requests going to both app servers instead of just to the one that handled the initial request.

Can you tell me what your JSESSIONID cookie looks like after you visit a page on one of your servers? I believe it's supposed to look like (random characters).(server ID) if sticky sessions are enabled, but perhaps the documentation is wrong on that point.

Here's my new workers2.properties.

[logger]
level=DEBUG

[config:]
file=/etc/apache2/workers2.properties
debug=0
debugEnv=0

[shm]
file=/tmp/mod_jk2_shm.file
size=1048576
debug=0
disabled=0

[workerEnv:]
info=Global server options
timing=1
debug=0

[lb:lb]
info=Default load balancer.
debug=1

[channel.socket:sgrimm1:8009]
port=8009
host=10.0.0.1
lb_factor=100

[channel.socket:sgrimm2:8009]
port=8009
host=10.0.0.2
lb_factor=100

[uri:/jsp/*]
info=JSPs.

[koreth]

Oh, I should add that I also tried replacing the hostnames with the IP addresses in the channel definitions, no effect that I can tell.

[ganton516]

My session ID looks like this:

E03QTrINyAW6tim2mxuZ1A**

The content of it changed pretty dramatically when I pulled my head out of my ass, and remembered to put the in the web.xml of the test application.

I would make my two channel definitions look like this, and make sure that each machine can ping the other by name.

[channel.socket:sgrimm1:8009]
port=8009
host=sgrimm1
lb_factor=100

[channel.socket:sgrimm2:8009]
port=8009
host=sgrimm2
lb_factor=100

Did you use the example cluster-service.xml file included in the "all" server with JBoss ?

Also keep in mind, that I am using 3.2.5, not 4.0, but I'm willing to bet it's mostly the same.

[koreth]

Just tried that, no change in behavior. The machines can definitely ping each other by name.

I wonder... Are you sure you're actually getting sticky sessions? If you reload a JSP or servlet a bunch of times, do the requests only ever go to one of the app servers? And when you clear the JSESSIONID cookie, do you sometimes get the other server?

I don't see how that JSESSIONID could work -- after I tried your latest suggestion, I browsed through the mod_jk2 source code. I'm looking at jk_requtil.c (the jk2_requtil_getSessionRoute function) and jk_worker_lb.c (the jk2_get_most_suitable_worker function). Unless I'm just completely misreading the code, it looks like if the JSESSIONID doesn't include a ".", the lb worker will round-robin the request among all its available servers (paying attention to the lb_factor, of course.) It doesn't look like there's any path through the code that would cause a JSESSIONID with no period to stick to a particular server -- apart from pulling out the jvmRoute value, the load balancer code doesn't even look at the session cookie.

Thanks for taking the time to keep up with this thread!

[koreth]

Got it!

It turns out there's one other thing you have to change. It's not documented in the 3.x clustering book or the "using mod_jk with JBoss" wiki page, but if you dig around in the wiki enough, you come across a link to "New JBoss3.2.6 Tomcat session replication" which has a brief mention of the extra configuration item.

Edit jbossweb-tomcat50.sar/META-INF/jboss-service.xml, search for "UseJK", and set its value to "true", like:

<attribute name="UseJK">true</attribute>

Once I did that and restarted, JBoss started putting the jvmRoute value at the end of my JSESSIONID cookies, and my sessions are now sticky. (The mod_jk2 configuration isn't an issue; it's the Tomcat configuration.)

Hope this helps someone else out there. If any of the JBoss developers are reading this, please please please add this to the 4.x clustering book so the next person doesn't waste a couple of days setting up what should be fairly basic functionality.