Mystery with Mod cluster

[gnanagurus]

Hello,

I am doing a performance test with a mod cluster setup. Please find the diagrams below. I have drewn it for your better understanding. Please correct me if my approach is wrong.

1. I am now using mod cluster in Linux box with 'worker' MPM.

2. My use case is simple camel applcation which gives http response for http requests.

3. I am using JMeter for sending the requests.

There is no difference in performance if I add or remove nodes. Please help.

Please help with your thoughts.

Regards

Guru

@gnanagurus

[jfclere]

Could please post that on https://community.jboss.org/message/802013
that is the same thing no?

[gnanagurus]

I thought the thread was closed since I flagged your response as 'Answered' .

Thats is why I made a new thread. Anyway, not a problem Jean. Sorry I will post to that thread.

Regards

Guru

[mbabacek]

Well, it is really impossible to help you without these crucial missing pieces of information:

• How many clients you used? I mean, was it 1 client issuing this many requests or thousands of clients?  (vertical/horizontal load)
• What was the load metric used and load reported on each node on mod_cluster manager console?
• Which nodes were serving requests?

You see, with sticky-sessions on true (default) and 1 client session only (regardless the number of requests this client sends), the balancer has no motivation to route requests to other nodes unless the currently elected node is responsive ok (based on the load-metric set on that node).

[rhusar]

Another thing is that the scenario is so simple that adding more nodes does not really help. Because the bottneck here is probably the network thus you wont see anything interesting in your benchmark. A benchmark needs to be well thought out in order to be useful.

So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlet https://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.java which generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot.

[gnanagurus]

Hello Michal,

1. As I have put in the diagram. I am using 5 JMeter instances, which means 5 clients. ( By the way each client has 5 threads running). So it sums up to 20 clients sending requests.

2. Currently I am using cpu+heap as load metric, each node shows 'Load:1'.

3. All four nodes are serving the request.

I understood your thoughts in sticky-sessions, can you give me a sample where I can configure sticky-sessions. Sorry I couldnt figure out. I hope that this may give a good result.

This is my mod cluster config from one of the JBoss AS node.

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">

          <mod-cluster-config proxy-list="10.200.208.124:6666" advertise="false">

                <dynamic-load-provider>

                    <load-metric type="cpu+heap" weight="3"/>

                </dynamic-load-provider>

         </mod-cluster-config>

</subsystem>

Regards

Guru

@gnanagurus

[mbabacek]

This is wrong:

                    <load-metric type="cpu+heap" weight="3"/>

you set it, for instance, as:

                    <load-metric type="cpu" weight="1"/>

                    <load-metric type="heap" weight="2"/>

By the way, Load:1 is a special value, you should be getting values >1 and <=100 at this point. Note that 100 menas no load. Smaller the fugure, higher th load.

[rhusar]

This is not even a valid metric. The model allows it in 7.1.1 but this is already fixed in newer versions of AS:

[standalone@localhost:9999 mod-cluster-config=configuration] :add-metric(type="cpu+heap", weight=123)
{
    "outcome" => "failed",
    "failure-description" => "JBAS014839: Invalid value cpu+heap for type; legal values are [heap, requests, sessions, busyness, send-traffic, mem, cpu, receive-traffic]",
    "rolled-back" => true
}

[gnanagurus]

Hi Radoslav and Michal,

I changed the metrics to 'busyness' now. But the performance results are logically reverse.

2 nodes = 2.4 ms

3 nodes = 3.3 ms

1. I have disabled the sticky-sessions

2. Previously the result pattern is same for any number of nodes, now its negative.

I have put my configurations below, such that you can correct if I am somewhere wrong.

mod_cluster system:

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">

      <mod-cluster-config proxy-list="10.200.208.229:6666" advertise="false" sticky-session="false">

           <dynamic-load-provider>

                <load-metric type="busyness"/>

           </dynamic-load-provider>

      </mod-cluster-config>

</subsystem>

MPM configuration in httpd-mpm.conf:

<IfModule mpm_worker_module>

                StartServers         15

                   MaxClients 128

                MinSpareThreads      75

              MaxSpareThreads 150

                ThreadsPerChild      64

              MaxRequestsPerChild   0

</IfModule>

httpd.conf:

Listen 10.200.208.229:6666

Maxcontext 10

Maxnode 40

Maxhost 40

<VirtualHost 10.200.208.229:6666>

<Location />

Order deny,allow

Deny from all

Allow from all

</Location>

KeepAliveTimeout 60

MaxKeepAliveRequests 100

ManagerBalancerName GuruBalancer

</VirtualHost>

Please share your thoughts.

Regards

Guru

@gnanagurus

[mbabacek]

Why did you disable sticky-sessions? Now, every time the request hits a worker that does not posses the proper session, session data must be retrieved from the cluster.

TL;DR: Having sticky-sessions is good unless you have a reason for disabling it.

With your busyness metric setting, bigger the number of threads being occupied serving requests, the bigger is load (smaller Load number.)

For instance, if you set your clients to call this servlet: BusyConnectorsLoadServlet.java each request will be delayed by milliseconds, so with more and more requests, you soon start to drain upon available threads, thus seeing a change in Load.

In my humble opinion:

Regarding the number you measured: 2.4 ms and  3.3 ms

Unless measured for hundreds of test runs for each configuration, these numbers are the same.

If you have Configuration A and Configuration B and you run each for several hundreds times and the average for Configuration A is bigger than average for Configuration B, then you can say something about the relation between these two.

If you run the test once with Configuration A and once with Configuration B, results being 2.4ms and 3.3ms, I would consider Configuration A and Configuration B being equal.

What I suggest:

Have more clients sending resources consuming requests (see AverageSystemLoadServlet.java for inspiration).

Have both your AS7 workers really bussy, melting down with heat.

Measure average response time for the test duration.

Run the same configiuration with 3 workers.

Have each configuration executed for many times and agregate average results.

I belive you shall then see some meaningful results.

[gnanagurus]

Hello Michal,

I did a sequence of tests, It sounds miserable again. Again, adding nodes increases the time taken.

These are the results obtained from a two and four node cluster.

Httpd.conf:

Listen 10.147.152.235:6666

Maxcontext 10

Maxnode 40

Maxhost 40

<VirtualHost 10.147.152.235:6666>

<Location />

Order deny,allow

Deny from all

Allow from all

</Location>

KeepAliveTimeout 60

MaxKeepAliveRequests 100

ManagerBalancerName mycluster

</VirtualHost>

httpd-mpm.conf:

<IfModule mpm_worker_module>

StartServers         15

MaxClients          128

MinSpareThreads      75

    MaxSpareThreads 150

ThreadsPerChild      64

    MaxRequestsPerChild   0

</IfModule>

standalone-ha.xml:

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">

<mod-cluster-config proxy-list="10.147.152.235:6666">

<dynamic-load-provider>

<load-metric type="busyness"/>

</dynamic-load-provider>

</mod-cluster-config>

</subsystem>

Please help.

-Guru

[rhusar]

I still think the test scenario is not suited for what you want to achieve. If the response time is barely a millisecond, the application server is not doing any work at all. Most of the time is spent on the network. Thus adding more nodes will not help this in any way -- it will only add some overhead of keeping multiple TCP connections to multiple servers. Your results seem to be correct to me.

[gnanagurus]

Hi Radoslav,

I understand. Can you please suggest a test scenario which can give a significant test pattern.

I get reminded of your earlier sugesstion, to do the test based on CPU addition.

"

So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlethttps://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.javawhich generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot. "

Can you suggest a clear idea which can help me.

Thank you.

- Guru

  @gnanagurus

[jfclere]

You are trying with a camel application, no? May be the application is the blocking point.

Try with a simple servlet.