5 Replies Latest reply on Nov 9, 2012 2:21 AM by Michal Karm Babacek

    load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

    Claudianus Adjai Newbie

      Hello -

      I am seeing some behavior that does not seem right to me with mod_cluster and I was wondering if this is how it is supposed to work.

       

      I am running three EAP 6 nodes fronted by apache 2 on a ubuntu 12.04 OS. Each of the servers (jboss and apache) has its own ip (virtual or physical). I am running mod_cluster 1.2 and EAP 6 for each of the jboss nodes. I am starting them with standalone-ha.xml

       

      I have configured apache to use various settings.

       

      Conf 1

       

      CreateBalancers 1

       

      <IfModule manager_module>

          Listen 192.168.10.25:9000

       

          <VirtualHost 192.168.10.25:9000>

          ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On

             ProxyPassReverse / balancer://mycluster

              ProxyPreserveHost On

       

          <Directory />

                Order deny,allow

            Deny from all

            Allow from 192.168.10.

              </Directory>

       

          <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from 192.168.10.

          </Location>

       

          ManagerBalancerName mycluster

              KeepAliveTimeout 300

              MaxKeepAliveRequests 0

              AdvertiseFrequency 5

              ServerAdvertise On

          EnableMCPMReceive

         

           </VirtualHost>

      </IfModule>

       

      Then I configured it like below

       

      Conf 2


      CreateBalancers 2

       

      <IfModule manager_module>

          Listen 192.168.10.25:9000

       

          <VirtualHost 192.168.10.25:9000>

          #ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On

              #ProxyPassReverse / balancer://mycluster

              #ProxyPreserveHost On

       

          <Directory />

                Order deny,allow

            Deny from all

            Allow from 192.168.10.

              </Directory>

       

          <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from 192.168.10.

          </Location>

       

          ManagerBalancerName mycluster

              KeepAliveTimeout 300

              MaxKeepAliveRequests 0

              AdvertiseFrequency 5

              ServerAdvertise On

          EnableMCPMReceive

         

           </VirtualHost>

      </IfModule>

       

      Both configurations work.

       

      My AS7 config for mod_cluster looks like below for each of the nodes:

      conf 1

       

      <subsystem xmlns="urn:jboss:domain:modcluster:1.1">

              <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.10.25:9000" proxy-url="/" advertise="true" excluded-contexts="ROOT,invoker,jbossws,juddi,console" auto-enable-contexts="true" stop-context-timeout="10" socket-timeout="20" sticky-session="true" sticky-session-remove="false" sticky-session-force="false" worker-timeout="-1" max-attempts="1" flush-packets="false" flush-wait="-1" ping="10" smax="-1" ttl="-1" node-timeout="-1" connector="ajp">

                  <simple-load-provider />

                </mod-cluster-config>

              </subsystem>

       

      conf 2

       

      <subsystem xmlns="urn:jboss:domain:modcluster:1.1">

                  <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.10.25:9000" proxy-url="/" advertise="true" excluded-contexts="ROOT,invoker,jbossws,juddi,console" auto-enable-contexts="true" stop-context-timeout="10" socket-timeout="20" sticky-session="true" sticky-session-remove="false" sticky-session-force="false" worker-timeout="-1" max-attempts="1" flush-packets="false" flush-wait="-1" ping="10" smax="-1" ttl="-1" node-timeout="-1" connector="ajp">

                      <dynamic-load-provider history="0" decay="2">

                          <load-metric type="busyness"/>

                          <load-metric type="sessions" weight="2" capacity="512"/>

                      </dynamic-load-provider>

                  </mod-cluster-config>

              </subsystem>

       

       

      I have tried various configurations and can never get apache to elect node 3.

      It always elect node 1 an then node 2 but almost never node 3.

      The only time I saw it elect node 3 was when I had two different web apps deployed and have been hitting the first one which was routed to node 1 or node 2 and then I tried to hit the second app and at that point node 3 was elected and that was the only time I ever saw it elected.

       

      It this how it is supposed to work?

      Why wouldn't it elect node 3? I tried various combination of dynamic load balancing policies and also the simple load provider to no avail.

       

      Has anyone else seeing this? What am I missing?

       

      Thanks,

       

      C

        • 1. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected
          Michal Karm Babacek Apprentice

          Hi Claudianus, I've done a quick test with a simple configuration and everything seems to be in order.

          In my 3 nodes setup, all of them have been utilized quite equally: elected-3-nodes.png

          Version being Mod_cluster 1.2.3.Final. Take a look:

           

          httpd

          <IfModule manager_module>
            Listen perf05:8888
            ManagerBalancerName qacluster
            <VirtualHost perf05:8888>
              <Location />
               Order deny,allow
               Deny from all
               # Just for playing...quite not suitable for production :-)
               Allow from all
              </Location>
          
              KeepAliveTimeout 60
              MaxKeepAliveRequests 0
              #ServerAdvertise on http://@IP@:port
              AdvertiseFrequency 5
              #AdvertiseSecurityKey secret
              AdvertiseGroup 224.0.1.105:6666
              EnableMCPMReceive
          
              <Location /mcm>
                 SetHandler mod_cluster-manager
                 Order deny,allow
                 Deny from all
                 # :-)
                 Allow from all
              </Location>
          
            </VirtualHost>
          </IfModule>
          

           

          worker node (AS7)

           

          <subsystem xmlns="urn:jboss:domain:modcluster:1.1">
              <mod-cluster-config advertise-socket="modcluster" advertise="true" sticky-session="true" sticky-session-force="false" sticky-session-remove="false" connector="ajp">
                  <dynamic-load-provider decay="2" history="10">
                  <!-- Metric #0--><load-metric weight="1" type="busyness"/>
                  </dynamic-load-provider>
              </mod-cluster-config>
          </subsystem>
          

           

          I would suggest sticking to defaults as long as you are not up to any special configuration. By adding one config attribute at a time, we might find out what is actually causing your problem. I suppose the load was the same on all the three boxes? Note that the busyness metric calculation is related to the number of available processors (something like 32 * Runtime.getRuntime().availableProcessors() is the default available thread pool...).

           

          HTH

          K

          1 of 1 people found this helpful
          • 2. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected
            Claudianus Adjai Newbie

            Hello Michael -

            Thank you for the quick reply. I will try 1.2.3 and see how it goes.

            I actuallystarted with the default and after mutiple trials I was not able to get node 3 to be elected and that is when I started playing with the configurations.

            As for the load yes it is the same or at least supposed to be the same. All 3 EAP 6 nodes and the apache server are running on the same Physiscal box. I just used virtual ip to isolate them. So the traits and processors configuration should be the same.

             

            I will report back once I redo the test with 1.2.3.

             

            BTW is 1.2.3.Final out? I only have 1.2.0.Final

             

            Regards,

             

            C

            • 3. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected
              Claudianus Adjai Newbie

              Hello Michael -

              I followed your advice and got the source from github (latest version I built was 1.2.2) and built it but mostly for the native which I then re-installed into apache2.

              I then updated my apache conf to look like this

               

              <IfModule manager_module>

                  Listen 192.168.10.25:9000

               

                  <VirtualHost 192.168.10.25:9000>

               

                  <Directory />

                        Order deny,allow

                    Deny from all

                    Allow from 192.168.10.

                      </Directory>

               

                  <Location /mod_cluster_manager>

                    SetHandler mod_cluster-manager

                    Order deny,allow

                    Deny from all

                    Allow from 192.168.10.

                  </Location>

               

                  ManagerBalancerName mycluster

                      KeepAliveTimeout 60

                      MaxKeepAliveRequests 0

                      AdvertiseFrequency 5

                      ServerAdvertise On

                  EnableMCPMReceive

                 

                   </VirtualHost>

              </IfModule>

               

              I also updated my worker nodes like below

               

              <subsystem xmlns="urn:jboss:domain:modcluster:1.1">

              <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.10.25:9000" proxy-url="/" advertise="true" excluded-contexts="ROOT,invoker,jbossws,juddi,console" auto-enable-contexts="true" stop-context-timeout="10" socket-timeout="20" sticky-session="true" sticky-session-remove="false" sticky-session-force="false" worker-timeout="-1" max-attempts="1" flush-packets="false" flush-wait="-1" ping="10" smax="-1" ttl="-1" node-timeout="-1" connector="ajp">

                              <dynamic-load-provider history="10" decay="2">

                                  <load-metric type="busyness"/>

                                  <!--<load-metric type="sessions" weight="2" capacity="512"/>-->

                              </dynamic-load-provider>

                          </mod-cluster-config>

               

                      </subsystem>

               

              However I am still seeing the behavior I described where node 3 is never elected.

              To test it have launched my app from 5 to 10 different browsers and most of the calls were routed to node 1 and occasionally to node 2.

              I don't know what else I am missing.

              • 4. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected
                Claudianus Adjai Newbie

                Hello Michael -

                it looks like it is working now or at least somewhat. Basically I removed the proxy-list from the mod-cluster-fig section of the worker config and restarted apache2 as well as restarting all three EAP 6 nodes. Now node 3 got elected and is actually the dominant node as it has got elected 8 times while node 1 only got elected 2 times and node 2 got elected once.

                I will keep testing but for now I will consider this issue resolved.

                 

                Regards,

                 

                C

                1 of 1 people found this helpful
                • 5. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected
                  Michal Karm Babacek Apprentice

                  IMHO you are not getting the picture right: It is not any kind of round robin here. The balancer is trying to send the request to the node that has all the session data on it in order to avoid costly look up on another node (session replication, <distributed/> allows this in you web app).

                   

                  Don't bother with

                   

                  has got elected 8 times while node 1 only got elected 2 times and node 2 got elected once

                  send a coupe of thousands of requests in many simultaneous sessions and then compare node utilization.