14 Replies Latest reply on Mar 15, 2006 2:17 PM by John Prout

    Http Session replication problems when NOT using sticky-sess

    John Prout Newbie

      I have a JBoss cluster, v4.0.3, with two nodes. Load balancing uses mod_jk, but without sticky-session.

      I am having problems when a session is invalidated and then a new set of attributes set in a new session (i.e. someone logs out, then logs in again as a different user).

      HttpSession sess = request.getSession(false);
      if (sess != null)
      {
       sess.invalidate();
      }
      sess = request.getSession(true);
      sess.setAttribute(Constants.USER_KEY, userInfo);
      

      My testing shows that the session that is invalidated isn't being removed on the other cluster node (not the node where this code is actually run). The result is that the new user who's logged in sometimes sees data from the previous user.

      Searching the JBoss site, this is the document which has the most information about what I'm seeing: http://www.jboss.org/developers/projects/jboss/tc5-clustering.html

      An extract follows:

      Note that because of the dual use of in-memory and distributed store sessions and because JBossCache does yet not support the notion of modification event only from remote node(s), the current usage of session replication must use sticky session. That is, only one dedicated clustering node will handle the session request unless there is a failover. This should not pose as a severe restriction since it will be rare indeed that sticky session is not used in production today. We will relax this restriction in the future release.


      My question is, does this mean that we have to use sticky-session, because otherwise, changes to a session on one cluster node won't necessarily be visible on the other cluster nodes?

      I have "snapshot-mode" set to "instant" and "cache-mode" set to "REPL_SYNC"

      Any help or suggestions much appreciated

      John

        • 1. Re: Http Session replication problems when NOT using sticky-
          Brian Stansberry Master

          We strongly encourage use of sticky sessions. But, that being said, if things are working properly what you're trying to do shoudn't give you the results your seeing

          What's your replication-granularity setting (SESSION or ATTRIBUTE)?

          • 2. Re: Http Session replication problems when NOT using sticky-
            John Prout Newbie

            We are trying to avoid using sticky-session because that would mean that we also have to use it on the two hardware load-ballancers in front of the cluster and web servers. The Operations group doesn't want to make this change...

            Replication-granularity is set in the jboss-web.xml file and we're not setting it, so the default must be used (is the default session?)

            John

            • 4. Re: Http Session replication problems when NOT using sticky-
              John Prout Newbie

              I now have a simple test application which reliably shows that if I attempt to remove either a session (using HttpSession.invalidate()) or a session attribute (using HttpSession.removeAttribute(String)), the change is not immediately made on the remote cluster node. I am testing to find out how long it is before the session is removed on the remote node.

              Should I open a bug report for this issue (supported by the test application) or is this not considered a bug?

              Can anyone suggest any changes I can make to fix this issue, or to shorten the time between when a session is invalidated on one cluster node and when the same session is invalidated on other nodes in the cluster? Switching to using sticky-session is the obvious fix, but that will cause other problems in our environment

              Thanks - John

              • 5. Re: Http Session replication problems when NOT using sticky-
                Ben Wang Master

                If you do removeAttribute the effect should be immediate though because you are using REPL_SYNC mode. How do you test it? And how long does it take?

                • 6. Re: Http Session replication problems when NOT using sticky-
                  John Prout Newbie

                  My test application uses struts, but the session attribute I'm using as a marker is not used by struts.

                  I have 2 JSP pages. Each displays the name of the host that served it, and then lists the name/value of all session attributes, like so:

                  <%
                   java.util.Enumeration enm = session.getAttributeNames();
                   while(enm.hasMoreElements())
                   {
                   String attr = (String)enm.nextElement();
                   out.println("Session variable: " + attr + " value: " + session.getAttribute(attr) + "<br />");
                   }
                  %>
                  

                  On one page I have a button that sets the marker session attribute, then another button (logout) that removes the attribute, then invalidates the session and forwards to the other page:

                   HttpSession session = request.getSession(false);
                   if(session != null)
                   {
                   session.removeAttribute("marker");
                   session.invalidate();
                   }
                  


                  The second page allows me to make another request and forwards back to the first page. The first page reliably redisplays the marker session attribute, which was removed, the the session invalidated (on the other cluster server) by the previous request

                  Since I am running this on a two server cluster, using (mod_jk) equal weighted, round-robin load balancing, the requests are served first by one server, then by the other. Session is invalidated on one server, then displayed by a JSP page which executes on the other server. I always display the name of the server which ran each page just to confirm where the page was executed.

                  John


                  • 7. Re: Http Session replication problems when NOT using sticky-
                    John Prout Newbie

                    I just found a parameter in jbossweb-tomcat55.sar/META-INF/jboss-service.xml which looked as though it should fix my problems:

                     <!-- A flag indicating if the local vm session value should be used if
                     it exists. When true, the existing vm local session values are used and
                     updates are replicated, but updates to the same session on other nodes
                     do not update the local session value. This mode is only useful for
                     failover. When false, the session value is obtained from the distributed
                     cache. This mode can be used with load balancing.
                     -->
                     <attribute name="UseLocalCache">true</attribute>
                    

                    This looked like it should solve my problems, but I changed the value to "false" and it didn't make any difference :-(

                    John

                    • 8. Re: Http Session replication problems when NOT using sticky-
                      Ben Wang Master

                      Actually, we do have test cases to test removeAttribute and invalidate during cluster mode. In the test, we use httpclient to invalidate the session on one node and retrive it from the second one (using the same session id).

                      So I'd suggest you do a simpler test. Simply, have two node cluster with sticky session. invalidate/remove on one node, then shut it down, it will failover to the other one. Retrieve it again, and you will see whether it works or not.

                      • 9. Re: Http Session replication problems when NOT using sticky-
                        John Prout Newbie

                        I'm sure that the scenario you describe will work, but the whole point of this thread is that I'm trying to make the cluster work when it's not using sticky-session. Is thre any way I can push the removeAttribute or invalidate to the other node without shutting down?

                        • 10. Re: Http Session replication problems when NOT using sticky-
                          Ben Wang Master

                          Actually our unit test case that uses httpclient does not assume sticky session at all, and it still works! So you should not need sticky session for this to work. I was just saying that so it is easier for you to troubleshoot your problem. You need to test your enviornment first.

                          • 11. Re: Http Session replication problems when NOT using sticky-
                            Sangeetha Pisharody Newbie

                            I have a similar problem.

                            We have a 3 node jboss cluster that does not use sticky sessions. I tested the following cases:

                            Case 1:
                            1. Node 1 sets an attribute "Attribute1" in the session.
                            2. Node 2 gets "Attribute 1" - SUCCESS
                            3. Node 2 sets an attribute "Attribute 2" in the session.
                            4. Node 3 gets "Attribute 2" - SUCCESS

                            Case 2:
                            1. Node 1 sets an attribute "Attribute1" in the session.
                            2. Node 2 gets "Attribute 1" - SUCCESS
                            3. Node 2 sets an attribute "Attribute 2" in the session.
                            4. Node 1 gets "Attribute 2" - FAILS (It gets a null value)

                            Any idea why this is happening? I am using jboss 3.2.6. I have
                            "snapshot-mode" set to "instant", "cache-mode" set to "REPL_SYNC",
                            replication granularity is default(session) and "UseLocalCache" set to false.

                            • 12. Re: Http Session replication problems when NOT using sticky-
                              Ben Wang Master

                              I'd suggest you try out the latest AS 4.0.3P1 to see if it still works for you.

                              • 13. Re: Http Session replication problems when NOT using sticky-
                                Brian Stansberry Master

                                Even better, hold out for 4.0.4.CR2 (which is scheduled for the end of the week) as I've made a number of improvements.

                                Particularly, the session invalidation issue that was the start of this thread is fixed: see http://jira.jboss.com/jira/browse/JBAS-2927.

                                But, all that being said, use sticky sessions!!

                                • 14. Re: Http Session replication problems when NOT using sticky-
                                  John Prout Newbie

                                  After some very careful testing of whether invalidate() and removeAttribute() are successfully replicated to cluster nodes other than the one where they were executed, we found that this reliably didn't work in a non-sticky-session configuration. The same tests worked correctly when we converted our test cluster to sticky-session, so we're in the process of converting our production cluster to use sticky-session.