12 Replies Latest reply on Mar 31, 2014 8:34 AM by rhusar

    Clustering without sticky sessions

    artbristol

      Is it possible to use JBoss behind a DNS-based load balancer (i.e. without sticky sessions)?

       

      When I set up two JBosses in a cluster on my local machine, and hit a simple test application (see https://github.com/artbristol/jbossexample) from two browser tabs simultaneously, frequently one browser tab times out, and you see the following in the logs

       

      JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of...

       

      All subsequent requests to that node are extremely slow. I've done some debugging - it seems one of the nodes is incorrectly hanging onto the session lock despite having finished serving the request. If you up the logging, you can see repeated instances of

       

      DEBUG [org.jboss.as.clustering] (http-/127.0.0.1:8180-1) [//] Received FAIL response from nodeMain/web -- reports lock is held by nodeMain/web

       

      This happens in AS 7.2.0, EAP 6.1.0.Final, and EAP 6.2.0.Beta

       

      Steps to reproduce:

       

      1. Build https://github.com/artbristol/jbossexample
      2. Unzip two copies of JBoss EAP 6.2 into separate directories and deploy jbossexample.war into standalone/deployments
      3. Start one with standalone.bat --server-config standalone-ha.xml
      4. Start the other with standalone.bat --server-config standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Djboss.node.name=plus100
      5. Start Firefox with one tab pointed at http://localhost:8080/jbossexample/get and the other at http://localhost:8180/jbossexample/get
      6. In Firefox, right click and choose 'Reload All Tabs' (You might need to try this 5-10 times before it fails)
        • 1. Re: Clustering without sticky sessions
          pferraro

          To use web session replication without sticky sessions, you'll need to make sure that your web session replicate synchronously.  Otherwise, you can't be sure that the node that serving a given session has the latest session data.

          To do this, edit the relevant infinispan subsystem section in standalone-ha.xml (or domain.xml).

          e.g.

          {code:xml}

          <cache-container name="web">

               <replicated-cache name="repl" mode="SYNC"/>

          </cache-container>

          {code}

          • 2. Re: Clustering without sticky sessions
            artbristol

            Hi Paul

             

            Thanks for the suggestion. I've just tried that, but the problem remains. Anything else I could try?

             

            Art

            • 3. Re: Clustering without sticky sessions
              rhusar

              Is it possible to use JBoss behind a DNS-based load balancer (i.e. without sticky sessions)?

              It seems to me that this is confusion.

               

              DNS-based load-balancing works in a way that it replies with a different A record (or set of records sorted in a way that last returned record is least, etc.) for a client. That being said, each client is going to communicate with one particular IP address (AS node) until a failure occurs (or A record ttl expires). So sticky sessions do not really come into play here.

              • 4. Re: Clustering without sticky sessions
                artbristol

                Hi, and thanks for the reply.

                 

                I agree with your description of how DNS load-balancing works. Yes, sticky sessions don't come into play.

                 

                But JBoss clustering doesn't seem to work in that configuration, in the occasional edge-cases in which client requests from the same session arrive at different endpoints over a short period (10-15 seconds) when the session lock is still held by one of the nodes. (This might happen, for example, when the A record ttl expires)

                 

                Are you able to try my steps to reproduce, to see if you have the same problem?

                • 5. Re: Re: Clustering without sticky sessions
                  rhusar

                  I cannot reproduce even with the test case you described (tried 50 times), neither with a script to test hopping from one node to another. Something like:

                   

                  [rhusar@rhusar script]$ cat multiple.sh
                  #!/bin/bash
                  echo "Hit CTRL+C to stop."
                  for (( ; ; ))
                  do
                    curl $1 -b cookiefile.txt -c cookiefile.txt
                    echo -n " ";
                    curl $2 -b cookiefile.txt -c cookiefile.txt
                    echo -n " ";
                  done
                  [rhusar@rhusar script]$ sh multiple.sh http://localhost:8080/clusterbench/session http://localhost:8180/clusterbench/session
                  Hit CTRL+C to stop.
                  0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
                  

                   

                  Tried with clusterbench and EAP 6.1.1.

                  1 of 1 people found this helpful
                  • 6. Re: Clustering without sticky sessions
                    artbristol

                    Hi Radoslav

                     

                    Thanks very much for trying... I know there were some concurrency fixes in EAP 6.1.1, which I don't have access to. Can you confirm that 6.1.0 is also working for you?

                     

                    Regards

                     

                    Art

                    • 7. Re: Clustering without sticky sessions
                      rhusar

                      Hm, ok so tried with jboss-eap-6.2.0.Beta.zip, everything works fine for me, neither of the scenario cause problems.

                      • 8. Re: Clustering without sticky sessions
                        rhusar

                        You may want to try to create a unit test that would reproduce the issue:

                         

                        https://github.com/wildfly/wildfly/tree/master/testsuite/integration/clust

                        • 9. Re: Clustering without sticky sessions
                          artbristol

                          OK, thanks again for the reply. I'll try it with Wildfly; if I can reproduce it there, I'll add an integration test.

                          • 10. Re: Clustering without sticky sessions
                            kylin

                            You must ensure sticky sessions are enabled in JBoss cluster, If sticky sessions are not being maintained, allowing the session to hop across the cluster rapidly, it will run into these lock issues.

                            • 11. Re: Clustering without sticky sessions
                              kylin

                              I don't know how DNS-based load balancer works, but JBoss cluster need sticky session enable, because the session to hop across the cluster rapidly to run into these lock issues.

                              • 12. Re: Clustering without sticky sessions
                                rhusar

                                Kylin Soong wrote:

                                 

                                I don't know how DNS-based load balancer works, but JBoss cluster need sticky session enable, because the session to hop across the cluster rapidly to run into these lock issues.

                                Well, in case od DNS-based balancing the client resolves the domain name into an IP address on the first request and the client reuses the result for the set duration to cache the DNS result. So there won't be any hopping.