4 Replies Latest reply on Feb 13, 2015 5:16 AM by rhusar

    mod_cluster and new nodes

    casele
    casele Feb 11, 2015 3:37 PM

      Using mod_cluster 1.2.1 in front of JBoss AS 7.1 servers.  We use busyness for the load metric.  A challenge that we are seeing is that when new nodes are added to a busy cluster the node receives a majority of the traffic for a few minutes, i assume because his elected numbers are low and his load is high.  Problem is however that often times this causes a DOS of that box and sometimes brings the JBoss server down.  As a result we've had to make a practice of adding many servers at once to limit this impact.

       

      Couple questions...first, any suggestions on ways to avoid this?  Additionally, it seems that the busyness metric calculates usage across all pools (HTTP and AJP).  Since we only use AJP, this returns false load numbers that never go below 50.

       

      Thoughts appreciated!

      -Casey

      • 392 Views
      • Tags:
        • 1. Re: mod_cluster and new nodes
          casele
          casele Feb 11, 2015 7:58 PM (in response to casele)

          I tried creating a custom LoadMetric that only uses the AJP connector and that helped, but the load number never gets to 0.  Reviewing the mod_cluster code for Apache HTTP, it appears that it would stop sending traffic to a node if it's load was 0...but DynamicLoadBalanceFactorProvider forces the number to be a 1 when I return a load that == capacity:

           

           

                  return 100 - Math.max(0, Math.min(load, 99));



          Any alternative ways of marking the node as "offline" when it needs to cool down?


          -Casey

            • Actions
          • 2. Re: mod_cluster and new nodes
            jfclere
            jfclere Feb 12, 2015 3:37 AM (in response to casele)

            The value 0 marks node as hot standby (message STATUS with load 0) and cause a failover of active sessions (Mod-Cluster_Management_Protocol).

              • Actions
            • 3. Re: mod_cluster and new nodes
              casele
              casele Feb 12, 2015 11:24 AM (in response to jfclere)

              Any recommendations or "best practices" to avoid a thunder herd when adding new nodes?

                • Actions
              • 4. Re: mod_cluster and new nodes
                rhusar
                rhusar Feb 13, 2015 5:16 AM (in response to casele)
                A challenge that we are seeing is that when new nodes are added to a busy cluster the node receives a majority of the traffic for a few minutes, i assume because his elected numbers are low and his load is high.  Problem is however that often times this causes a DOS of that box and sometimes brings the JBoss server down.  As a result we've had to make a practice of adding many servers at once to limit this impact.

                Unfortunately this is a very important feature to be missing.

                 

                Created [MODCLUSTER-449] Implement ramp-up when starting new nodes - JBoss Issue Tracker

                  • Actions