13 Replies Latest reply on Mar 26, 2013 8:53 AM by Jean-Frederic Clere

    Mystery with Mod cluster

    Gnanaguru Sattanathan Novice

      Hello,

       

      I am doing a performance test with a mod cluster setup. Please find the diagrams below. I have drewn it for your better understanding. Please correct me if my approach is wrong.

       

      1. I am now using mod cluster in Linux box with 'worker' MPM.

      2. My use case is simple camel applcation which gives http response for http requests.

      3. I am using JMeter for sending the requests.

       

      There is no difference in performance if I add or remove nodes. Please help.

       

      mod cluster 1.png

      mod cluster 2.png

       

       

      Please help with your thoughts.

       

      Regards

      Guru

      @gnanagurus

        • 2. Re: Mystery with Mod cluster
          Gnanaguru Sattanathan Novice

          I thought the thread was closed since I flagged your response as 'Answered' .

           

          Thats is why I made a new thread. Anyway, not a problem Jean. Sorry I will post to that thread.

           

          Regards

          Guru

          • 3. Re: Mystery with Mod cluster
            Michal Karm Babacek Apprentice

            Well, it is really impossible to help you without these crucial missing pieces of information:

            • How many clients you used? I mean, was it 1 client issuing this many requests or thousands of clients?  (vertical/horizontal load)
            • What was the load metric used and load reported on each node on mod_cluster manager console?
            • Which nodes were serving requests?

             

            You see, with sticky-sessions on true (default) and 1 client session only (regardless the number of requests this client sends), the balancer has no motivation to route requests to other nodes unless the currently elected node is responsive ok (based on the load-metric set on that node).

            • 4. Re: Mystery with Mod cluster
              Radoslav Husar Master

              Another thing is that the scenario is so simple that adding more nodes does not really help. Because the bottneck here is probably the network thus you wont see anything interesting in your benchmark. A benchmark needs to be well thought out in order to be useful.

               

              So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlet https://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.java which generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot.

              • 5. Re: Mystery with Mod cluster
                Gnanaguru Sattanathan Novice

                Hello Michal,

                 

                 

                1. As I have put in the diagram. I am using 5 JMeter instances, which means 5 clients. ( By the way each client has 5 threads running). So it sums up to 20 clients sending requests.

                 

                 

                2. Currently I am using cpu+heap as load metric, each node shows 'Load:1'.

                 

                 

                3. All four nodes are serving the request.

                 

                 

                I understood your thoughts in sticky-sessions, can you give me a sample where I can configure sticky-sessions. Sorry I couldnt figure out. I hope that this may give a good result.

                 

                This is my mod cluster config from one of the JBoss AS node.

                 

                <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                          <mod-cluster-config proxy-list="10.200.208.124:6666" advertise="false">

                                <dynamic-load-provider>

                                    <load-metric type="cpu+heap" weight="3"/>

                                </dynamic-load-provider>

                         </mod-cluster-config>

                </subsystem>

                 

                 

                 

                Regards

                Guru

                @gnanagurus

                • 6. Re: Mystery with Mod cluster
                  Michal Karm Babacek Apprentice

                  This is wrong:

                                      <load-metric type="cpu+heap" weight="3"/>

                  you set it, for instance, as:

                                      <load-metric type="cpu" weight="1"/>

                                      <load-metric type="heap" weight="2"/>

                  By the way, Load:1 is a special value, you should be getting values >1 and <=100 at this point. Note that 100 menas no load. Smaller the fugure, higher th load.

                  • 7. Re: Mystery with Mod cluster
                    Radoslav Husar Master

                    This is not even a valid metric. The model allows it in 7.1.1 but this is already fixed in newer versions of AS:

                     

                    [standalone@localhost:9999 mod-cluster-config=configuration] :add-metric(type="cpu+heap", weight=123)
                    {
                        "outcome" => "failed",
                        "failure-description" => "JBAS014839: Invalid value cpu+heap for type; legal values are [heap, requests, sessions, busyness, send-traffic, mem, cpu, receive-traffic]",
                        "rolled-back" => true
                    }
                    
                    • 8. Re: Mystery with Mod cluster
                      Gnanaguru Sattanathan Novice

                      Hi Radoslav and Michal,

                       

                      I changed the metrics to 'busyness' now. But the performance results are logically reverse.

                       

                      2 nodes = 2.4 ms

                      3 nodes = 3.3 ms

                       

                      1. I have disabled the sticky-sessions

                      2. Previously the result pattern is same for any number of nodes, now its negative.

                       

                      I have put my configurations below, such that you can correct if I am somewhere wrong.

                       

                      mod_cluster system:

                       

                      <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                            <mod-cluster-config proxy-list="10.200.208.229:6666" advertise="false" sticky-session="false">

                                 <dynamic-load-provider>

                                      <load-metric type="busyness"/>

                                 </dynamic-load-provider>

                            </mod-cluster-config>

                      </subsystem>

                       

                      MPM configuration in httpd-mpm.conf:

                       

                      <IfModule mpm_worker_module>

                                      StartServers         15

                                         MaxClients 128

                                      MinSpareThreads      75

                                    MaxSpareThreads 150

                                      ThreadsPerChild      64

                                    MaxRequestsPerChild   0

                      </IfModule>

                       

                      httpd.conf:

                       

                      Listen 10.200.208.229:6666

                      Maxcontext 10

                      Maxnode 40

                      Maxhost 40

                      <VirtualHost 10.200.208.229:6666>

                      <Location />

                      Order deny,allow

                      Deny from all

                      Allow from all

                      </Location>

                      KeepAliveTimeout 60

                      MaxKeepAliveRequests 100

                      ManagerBalancerName GuruBalancer

                      </VirtualHost>

                       

                      Please share your thoughts.

                       

                       

                      Regards

                      Guru

                      @gnanagurus

                      • 9. Re: Mystery with Mod cluster
                        Michal Karm Babacek Apprentice

                        Why did you disable sticky-sessions? Now, every time the request hits a worker that does not posses the proper session, session data must be retrieved from the cluster.

                        TL;DR: Having sticky-sessions is good unless you have a reason for disabling it.

                         

                        With your busyness metric setting, bigger the number of threads being occupied serving requests, the bigger is load (smaller Load number.)

                        For instance, if you set your clients to call this servlet: BusyConnectorsLoadServlet.java each request will be delayed by milliseconds, so with more and more requests, you soon start to drain upon available threads, thus seeing a change in Load.

                         

                        In my humble opinion:

                        Regarding the number you measured: 2.4 ms and  3.3 ms

                        Unless measured for hundreds of test runs for each configuration, these numbers are the same.

                        If you have Configuration A and Configuration B and you run each for several hundreds times and the average for Configuration A is bigger than average for Configuration B, then you can say something about the relation between these two.

                        If you run the test once with Configuration A and once with Configuration B, results being 2.4ms and 3.3ms, I would consider Configuration A and Configuration B being equal.

                         

                        What I suggest:

                        Have more clients sending resources consuming requests (see AverageSystemLoadServlet.java for inspiration).

                        Have both your AS7 workers really bussy, melting down with heat.

                        Measure average response time for the test duration.

                         

                        Run the same configiuration with 3 workers.

                         

                        Have each configuration executed for many times and agregate average results.

                         

                        I belive you shall then see some meaningful results.

                        • 10. Re: Mystery with Mod cluster
                          Gnanaguru Sattanathan Novice

                          Hello Michal,

                           

                          I did a sequence of tests, It sounds miserable again. Again, adding nodes increases the time taken.

                           

                          These are the results obtained from a two and four node cluster.

                           

                          mod_cluster.png

                           

                          Httpd.conf:

                           

                          Listen 10.147.152.235:6666

                          Maxcontext 10

                          Maxnode 40

                          Maxhost 40

                          <VirtualHost 10.147.152.235:6666>

                          <Location />

                          Order deny,allow

                          Deny from all

                          Allow from all

                          </Location>

                          KeepAliveTimeout 60

                          MaxKeepAliveRequests 100

                          ManagerBalancerName mycluster

                          </VirtualHost>

                           

                          httpd-mpm.conf:

                           

                          <IfModule mpm_worker_module>

                          StartServers         15

                          MaxClients          128

                          MinSpareThreads      75

                              MaxSpareThreads 150

                          ThreadsPerChild      64

                              MaxRequestsPerChild   0

                          </IfModule>

                           

                          standalone-ha.xml:

                           

                          <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                          <mod-cluster-config proxy-list="10.147.152.235:6666">

                          <dynamic-load-provider>

                          <load-metric type="busyness"/>

                          </dynamic-load-provider>

                          </mod-cluster-config>

                          </subsystem>

                           

                          Please help.

                           

                          -Guru

                          • 11. Re: Mystery with Mod cluster
                            Radoslav Husar Master

                            I still think the test scenario is not suited for what you want to achieve. If the response time is barely a millisecond, the application server is not doing any work at all. Most of the time is spent on the network. Thus adding more nodes will not help this in any way -- it will only add some overhead of keeping multiple TCP connections to multiple servers. Your results seem to be correct to me.

                            • 12. Re: Mystery with Mod cluster
                              Gnanaguru Sattanathan Novice

                              Hi Radoslav,

                               

                              I understand. Can you please suggest a test scenario which can give a significant test pattern.

                               

                              I get reminded of your earlier sugesstion, to do the test based on CPU addition.

                               

                              "

                              So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlethttps://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.javawhich generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot. "

                               

                              Can you suggest a clear idea which can help me.

                               

                              Thank you.

                               

                              - Guru

                                @gnanagurus

                              • 13. Re: Mystery with Mod cluster
                                Jean-Frederic Clere Master

                                You are trying with a camel application, no? May be the application is the blocking point.

                                Try with a simple servlet.