-
1. Re: Mystery with Mod cluster
jfclere Mar 14, 2013 6:27 AM (in response to gnanagurus)Could please post that on https://community.jboss.org/message/802013
that is the same thing no? -
2. Re: Mystery with Mod cluster
gnanagurus Mar 14, 2013 6:37 AM (in response to jfclere)I thought the thread was closed since I flagged your response as 'Answered' .
Thats is why I made a new thread. Anyway, not a problem Jean. Sorry I will post to that thread.
Regards
Guru
-
3. Re: Mystery with Mod cluster
mbabacek Mar 14, 2013 6:43 AM (in response to gnanagurus)Well, it is really impossible to help you without these crucial missing pieces of information:
- How many clients you used? I mean, was it 1 client issuing this many requests or thousands of clients? (vertical/horizontal load)
- What was the load metric used and load reported on each node on mod_cluster manager console?
- Which nodes were serving requests?
You see, with sticky-sessions on true (default) and 1 client session only (regardless the number of requests this client sends), the balancer has no motivation to route requests to other nodes unless the currently elected node is responsive ok (based on the load-metric set on that node).
-
4. Re: Mystery with Mod cluster
rhusar Mar 14, 2013 7:23 AM (in response to mbabacek)Another thing is that the scenario is so simple that adding more nodes does not really help. Because the bottneck here is probably the network thus you wont see anything interesting in your benchmark. A benchmark needs to be well thought out in order to be useful.
So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlet https://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.java which generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot.
-
5. Re: Mystery with Mod cluster
gnanagurus Mar 14, 2013 7:57 AM (in response to mbabacek)Hello Michal,
1. As I have put in the diagram. I am using 5 JMeter instances, which means 5 clients. ( By the way each client has 5 threads running). So it sums up to 20 clients sending requests.
2. Currently I am using cpu+heap as load metric, each node shows 'Load:1'.
3. All four nodes are serving the request.
I understood your thoughts in sticky-sessions, can you give me a sample where I can configure sticky-sessions. Sorry I couldnt figure out. I hope that this may give a good result.
This is my mod cluster config from one of the JBoss AS node.
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config proxy-list="10.200.208.124:6666" advertise="false">
<dynamic-load-provider>
<load-metric type="cpu+heap" weight="3"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
Regards
Guru
@gnanagurus
-
6. Re: Mystery with Mod cluster
mbabacek Mar 14, 2013 8:05 AM (in response to gnanagurus)This is wrong:
<load-metric type="cpu+heap" weight="3"/>
you set it, for instance, as:
<load-metric type="cpu" weight="1"/>
<load-metric type="heap" weight="2"/>
By the way, Load:1 is a special value, you should be getting values >1 and <=100 at this point. Note that 100 menas no load. Smaller the fugure, higher th load.
-
7. Re: Mystery with Mod cluster
rhusar Mar 14, 2013 10:20 AM (in response to mbabacek)This is not even a valid metric. The model allows it in 7.1.1 but this is already fixed in newer versions of AS:
[standalone@localhost:9999 mod-cluster-config=configuration] :add-metric(type="cpu+heap", weight=123) { "outcome" => "failed", "failure-description" => "JBAS014839: Invalid value cpu+heap for type; legal values are [heap, requests, sessions, busyness, send-traffic, mem, cpu, receive-traffic]", "rolled-back" => true }
-
8. Re: Mystery with Mod cluster
gnanagurus Mar 15, 2013 1:32 AM (in response to rhusar)Hi Radoslav and Michal,
I changed the metrics to 'busyness' now. But the performance results are logically reverse.
2 nodes = 2.4 ms
3 nodes = 3.3 ms
1. I have disabled the sticky-sessions
2. Previously the result pattern is same for any number of nodes, now its negative.
I have put my configurations below, such that you can correct if I am somewhere wrong.
mod_cluster system:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config proxy-list="10.200.208.229:6666" advertise="false" sticky-session="false">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
MPM configuration in httpd-mpm.conf:
<IfModule mpm_worker_module>
StartServers 15
MaxClients 128
MinSpareThreads 75
MaxSpareThreads 150
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
httpd.conf:
Listen 10.200.208.229:6666
Maxcontext 10
Maxnode 40
Maxhost 40
<VirtualHost 10.200.208.229:6666>
<Location />
Order deny,allow
Deny from all
Allow from all
</Location>
KeepAliveTimeout 60
MaxKeepAliveRequests 100
ManagerBalancerName GuruBalancer
</VirtualHost>
Please share your thoughts.
Regards
Guru
@gnanagurus
-
9. Re: Mystery with Mod cluster
mbabacek Mar 15, 2013 7:00 AM (in response to gnanagurus)Why did you disable sticky-sessions? Now, every time the request hits a worker that does not posses the proper session, session data must be retrieved from the cluster.
TL;DR: Having sticky-sessions is good unless you have a reason for disabling it.
With your busyness metric setting, bigger the number of threads being occupied serving requests, the bigger is load (smaller Load number.)
For instance, if you set your clients to call this servlet: BusyConnectorsLoadServlet.java each request will be delayed by milliseconds, so with more and more requests, you soon start to drain upon available threads, thus seeing a change in Load.
In my humble opinion:
Regarding the number you measured: 2.4 ms and 3.3 ms
Unless measured for hundreds of test runs for each configuration, these numbers are the same.
If you have Configuration A and Configuration B and you run each for several hundreds times and the average for Configuration A is bigger than average for Configuration B, then you can say something about the relation between these two.
If you run the test once with Configuration A and once with Configuration B, results being 2.4ms and 3.3ms, I would consider Configuration A and Configuration B being equal.
What I suggest:
Have more clients sending resources consuming requests (see AverageSystemLoadServlet.java for inspiration).
Have both your AS7 workers really bussy, melting down with heat.
Measure average response time for the test duration.
Run the same configiuration with 3 workers.
Have each configuration executed for many times and agregate average results.
I belive you shall then see some meaningful results.
-
10. Re: Mystery with Mod cluster
gnanagurus Mar 25, 2013 4:09 AM (in response to mbabacek)Hello Michal,
I did a sequence of tests, It sounds miserable again. Again, adding nodes increases the time taken.
These are the results obtained from a two and four node cluster.
Httpd.conf:
Listen 10.147.152.235:6666
Maxcontext 10
Maxnode 40
Maxhost 40
<VirtualHost 10.147.152.235:6666>
<Location />
Order deny,allow
Deny from all
Allow from all
</Location>
KeepAliveTimeout 60
MaxKeepAliveRequests 100
ManagerBalancerName mycluster
</VirtualHost>
httpd-mpm.conf:
<IfModule mpm_worker_module>
StartServers 15
MaxClients 128
MinSpareThreads 75
MaxSpareThreads 150
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
standalone-ha.xml:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config proxy-list="10.147.152.235:6666">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
Please help.
-Guru
-
11. Re: Mystery with Mod cluster
rhusar Mar 25, 2013 9:33 AM (in response to gnanagurus)I still think the test scenario is not suited for what you want to achieve. If the response time is barely a millisecond, the application server is not doing any work at all. Most of the time is spent on the network. Thus adding more nodes will not help this in any way -- it will only add some overhead of keeping multiple TCP connections to multiple servers. Your results seem to be correct to me.
-
12. Re: Mystery with Mod cluster
gnanagurus Mar 26, 2013 12:52 AM (in response to rhusar)Hi Radoslav,
I understand. Can you please suggest a test scenario which can give a significant test pattern.
I get reminded of your earlier sugesstion, to do the test based on CPU addition.
"
So for example: If you are adding more nodes, you are increasing CPU capacity among other things. You can try a test where you leverage that added CPU power. You can get inspired by this servlethttps://github.com/modcluster/mod_cluster/blob/master/demo/server/src/main/java/org/jboss/modcluster/demo/servlet/SystemLoadServlet.javawhich generates CPU load on the machine using an naughty loop. Then you will see that adding more nodes improves the response times a lot. "
Can you suggest a clear idea which can help me.
Thank you.
- Guru
@gnanagurus
-
13. Re: Mystery with Mod cluster
jfclere Mar 26, 2013 8:53 AM (in response to gnanagurus)You are trying with a camel application, no? May be the application is the blocking point.
Try with a simple servlet.