6 Replies Latest reply on Nov 12, 2004 3:15 PM by Steven Grimm

    Can't get sticky sessions to work

    Steven Grimm Newbie

      I have a cluster of two JBoss 4.0.1RC1 servers sitting behind an Apache 2 server running mod_jk2. The Apache server is doing load balancing and failover, but so far I have been completely unable to get session stickiness to work. The JSESSIONID cookie never includes the jvmRoute value.

      I've modified my jbossweb-tomcat50.sar/server.xml file's Engine tag:

      host 1:

      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="sgrimm1">

      host 2:
      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="sgrimm2">


      My application's web.xml has the "distributable" tag.

      The two hosts see each other in the cluster; I get log messages at startup like

      14:37:20,056 INFO [DefaultPartition] New cluster view for partition DefaultPartition: 9 ([10.0.0.1:1099, 10.0.0.2:1099] delta: 0)
      14:37:20,067 INFO [DefaultPartition] I am (null) received membershipChanged event:
      14:37:20,071 INFO [DefaultPartition] Dead members: 0 ([])
      14:37:20,076 INFO [DefaultPartition] New Members : 0 ([])
      14:37:20,080 INFO [DefaultPartition] All Members : 2 ([10.0.0.1:1099, 10.0.0.2:1099])


      I tried running a network sniffer on the traffic between Apache and JBoss, and I see that JBoss is sending back a JSESSIONID cookie that doesn't include the jvmRoute value. So I assume it's not a problem with the configuration on the Apache side. Or is mod_jk2 supposed to be the thing that appends the jvmRoute to the session cookie? Here's my workers2.properties just in case it's relevant.

      [shm]
      file=/tmp/mod_jk2_shm.file
      size=1048576
      
      [channel.socket:sgrimm1:8009]
      [channel.socket:sgrimm2:8009]
      
      [ajp13:sgrimm1:8009]
      channel=channel.socket:sgrimm1:8009
      tomcatId=sgrimm1
      
      [ajp13:sgrimm2:8009]
      channel=channel.socket:sgrimm2:8009
      tomcatId=sgrimm2
      
      [status:status]
      
      [lb:lb]
      stickySession=1
      
      [uri:/jsp/*]
      worker=lb
      
      [uri:/jmx-console/*]
      worker=lb


      I originally had "[lb]" instead of "[lb:lb]" but it made no difference.

      In attempting to diagnose the problem, I tried adding a debug logger to the Tomcat server.xml, using the existing logger as a guide:

      <Logger className="org.jboss.web.tomcat.Log4jLogger"
       verbosityLevel="DEBUG"
       category="org.jboss.web"/>


      I don't get any debug output, but I do get this message:

      13:35:30,262 INFO [STDOUT] Can't find resource org.jboss.web.tomcat.session.LocalStrings org.jboss.web.tomcat.tc5.WebCtxLoader$ENCLoader@f3387
      13:35:30,265 INFO [STDOUT] [Ljava.net.URL;@1e16028


      (It looks like there is a LocalStrings.properties in org.jboss.web.tomcat.tc5.session, but not in org.jboss.web.tomcat.session.)

      Any thoughts on how I can diagnose this, or better yet, fix it?

        • 1. Re: Can't get sticky sessions to work
          Wayne Hannam Newbie

          koreth:

          I JUST got all this working, the one big difference I can see between your setup and mine, is that when running the tomcats on separate machines, you do NOT need the jvmRoute, and tomcatID attributes, each jvm only has it's own route.

          Here is my workers2.properties file, adjust yours accordingly.

          I will put a watch on this thread, if this works, or doesn't, please let me know, I know how frustrating this is.

          [logger]
          level=DEBUG

          [config:]
          file=/data/apache/conf/workers2.properties
          debug=0
          debugEnv=0

          [shm]
          info=Scoreboard. Required for reconfiguration and status with multiprocess servers
          file=/var/log/apache/mod_jk2.shm
          size=1000000
          debug=0
          disabled=0

          [workerEnv:]
          info=Global server options
          timing=1
          debug=0

          [lb:lb]
          info=Default load balancer.
          debug=1

          # appserv1
          [channel.socket:10.0.0.1:8009]
          port=8009
          host=10.0.0.1
          lb_factor=100

          # appserv2
          [channel.socket:10.0.0.2:8009]
          port=8009
          host=10.0.0.2
          lb_factor=100

          [uri:/wayneh/*]
          info=Waynes Website

          • 2. Re: Can't get sticky sessions to work
            Steven Grimm Newbie

            Thanks for the suggestion. I modified my workers2.properties to look like yours, but still no stickiness. Apache still happily talks to both of the app servers, but if I open a browser, go to one of my JSPs, and repeatedly reload the page, I see the requests going to both app servers instead of just to the one that handled the initial request.

            Can you tell me what your JSESSIONID cookie looks like after you visit a page on one of your servers? I believe it's supposed to look like (random characters).(server ID) if sticky sessions are enabled, but perhaps the documentation is wrong on that point.

            Here's my new workers2.properties.

            [logger]
            level=DEBUG
            
            [config:]
            file=/etc/apache2/workers2.properties
            debug=0
            debugEnv=0
            
            [shm]
            file=/tmp/mod_jk2_shm.file
            size=1048576
            debug=0
            disabled=0
            
            [workerEnv:]
            info=Global server options
            timing=1
            debug=0
            
            [lb:lb]
            info=Default load balancer.
            debug=1
            
            [channel.socket:sgrimm1:8009]
            port=8009
            host=10.0.0.1
            lb_factor=100
            
            [channel.socket:sgrimm2:8009]
            port=8009
            host=10.0.0.2
            lb_factor=100
            
            [uri:/jsp/*]
            info=JSPs.


            • 3. Re: Can't get sticky sessions to work
              Steven Grimm Newbie

              Oh, I should add that I also tried replacing the hostnames with the IP addresses in the channel definitions, no effect that I can tell.

              • 4. Re: Can't get sticky sessions to work
                Wayne Hannam Newbie

                My session ID looks like this:

                E03QTrINyAW6tim2mxuZ1A**

                The content of it changed pretty dramatically when I pulled my head out of my ass, and remembered to put the in the web.xml of the test application.

                I would make my two channel definitions look like this, and make sure that each machine can ping the other by name.

                [channel.socket:sgrimm1:8009]
                port=8009
                host=sgrimm1
                lb_factor=100

                [channel.socket:sgrimm2:8009]
                port=8009
                host=sgrimm2
                lb_factor=100

                Did you use the example cluster-service.xml file included in the "all" server with JBoss ?

                Also keep in mind, that I am using 3.2.5, not 4.0, but I'm willing to bet it's mostly the same.

                • 5. Re: Can't get sticky sessions to work
                  Steven Grimm Newbie

                  Just tried that, no change in behavior. The machines can definitely ping each other by name.

                  I wonder... Are you sure you're actually getting sticky sessions? If you reload a JSP or servlet a bunch of times, do the requests only ever go to one of the app servers? And when you clear the JSESSIONID cookie, do you sometimes get the other server?

                  I don't see how that JSESSIONID could work -- after I tried your latest suggestion, I browsed through the mod_jk2 source code. I'm looking at jk_requtil.c (the jk2_requtil_getSessionRoute function) and jk_worker_lb.c (the jk2_get_most_suitable_worker function). Unless I'm just completely misreading the code, it looks like if the JSESSIONID doesn't include a ".", the lb worker will round-robin the request among all its available servers (paying attention to the lb_factor, of course.) It doesn't look like there's any path through the code that would cause a JSESSIONID with no period to stick to a particular server -- apart from pulling out the jvmRoute value, the load balancer code doesn't even look at the session cookie.

                  Thanks for taking the time to keep up with this thread!

                  • 6. Re: Can't get sticky sessions to work
                    Steven Grimm Newbie

                    Got it!

                    It turns out there's one other thing you have to change. It's not documented in the 3.x clustering book or the "using mod_jk with JBoss" wiki page, but if you dig around in the wiki enough, you come across a link to "New JBoss3.2.6 Tomcat session replication" which has a brief mention of the extra configuration item.

                    Edit jbossweb-tomcat50.sar/META-INF/jboss-service.xml, search for "UseJK", and set its value to "true", like:

                    <attribute name="UseJK">true</attribute>


                    Once I did that and restarted, JBoss started putting the jvmRoute value at the end of my JSESSIONID cookies, and my sessions are now sticky. (The mod_jk2 configuration isn't an issue; it's the Tomcat configuration.)

                    Hope this helps someone else out there. If any of the JBoss developers are reading this, please please please add this to the 4.x clustering book so the next person doesn't waste a couple of days setting up what should be fairly basic functionality.