Performance of an application works in cluster is lower than in a single server?

[minhld]

Hi all,

I have deployed my web service application on a JBoss cluster with three nodes. I used Apache HTTP Server, with mod_jk module to dispatch requests to each node in the cluster.

I wrote a simple client application to test the web service by calling a function in 10000 times, and I run the test in two cases: first with cluster through Apache, and the second connects directly to the JBoss server in one node. I noticed in the first case, the client application runs longer than in the second case. I guess the reason is Apache server waits until it gets a response from a node in the cluster and send back to client before sending another request to cluster, plus with time to select node and transfer requests between Apache and node, make it worse than the second case.

I want to deploy my web service application on JBoss cluster because I expect it would leverage the performance by alternately invoking each node in an asynchronous mode but it didn't work that way. I also tried to re-build the Apache server with MPM worker and configure the mod_jk but it didn't work. Honestly I'm not sure where to tune, in JBoss cluster, Apache or mod_jk.

If there is anyone got the same problem and already solved, please share your opinion and solution. Thank you!

[rhusar]

Hi Minhld,

just in pure theory, clustered application compared to standalone single node application must be slower (eg. 2 clustered nodes compared to 2 standalone modes). This is because there are numerous other things that clustered application needs to handle that the single node does not. There is replication between servers which has to serialize, send over the network and deserialize sessions which in the end give you additional features like high availability (HA). You might want to consider using nonclustered enviropnment, just run multiple nodes that do not talk to each other, but then you dont get HA.

The question is "how much slower" is bad.

Rado

[minhld]

Hi Radoslav,

Thank you very much for your quick answer.

I agree that if Apache server works that way in synchronous mode with JBoss cluster, the clustered application will be slower. I would like to know if there is a way to configure on Apache side or JBoss cluster side, so Apache server can asynchronously send requests to the nodes in cluster and correspondingly get the responses and send back to client. Because by this way, we can speed up the performance. But I'm not sure whether it is possible or not.

The reason I still do believe in is because of the introduction of JBoss Cluster: "... Clustering is crucial for scalable enterprise applications, as you can improve performance by adding more nodes to the cluster..." (http://docs.jboss.org/jbossclustering/cluster_guide/5.1/html-single/index.html#clustering-intro.chapt). I guess there must be a way that I didn't know.

Do you have any idea for this? Thank you!

[minhld]

Hi all,

Is there anyone implemented the JBoss Cluster with the feature that was introduced: "... Clustering is crucial for scalable enterprise applications, as you can improve performance by adding more nodes to the cluster...".

It seems like using Apache HTTP Server is not the right solution since its threading model is not asynchronous. I'm eagerly looking for a solution for this...

Any suggestion, any clue will be much appreciated! Thank you!

[rhusar]

We really need to explain first the

so Apache server can asynchronously send requests to the nodes in cluster and correspondingly get the responses and send back to client

What do you mean here? Apache as far as I know doesnt work like that as there isnt really a reason for it. A request is handled in a single thread, the more requests you have the more threads you will spawn.

Most people use Apache HTTPd with mod_jk and mod_cluster without issues :-)

[minhld]

Hi Radoslav,

Thank you for your answer. My english is bad, I mean I need a solution such as

Apache server can asynchronously send requests to the nodes in cluster and correspondingly get the responses and send back to client

I'm going to attach a picture that figures out my understanding about Apache and what I expected. Please correct me if I'm wrong.

I think it would describe better any of my words . The reason I want the second model is: in my application each request may consume time a lot, and if there are two separated clients, one must wait until the other has been completed. I will try mod_cluster, but please let me know whether mod_cluster can do as in the second model or not? Thank you!

[wdfink]

As I understand you wondering about the longer time for one request when you call with one Thread from client.

This is a normal behaviour.

1)

If you call JBoss direct you have a connection between your client and JBoss.

2)

If you call apache and the request is routed to a JBoss (or load-balanced to one of a JBoss cluster) there is an aditional latency.

What you know about Apache is correct, the request wait for the result and handle the response.

The benefit of xuch clustering is NOT to minimize the duration but to increase the scalability.

Try the test and increase the number of clients and you will see that you will reach a point where one JBoss instance is overloaded and you can not increase the throughput.

If you use apache with two (or more) JBoss instances you will be slower if you have one client, but the scalability will be better and you can have more load.

[rhusar]

Regarding the picture,

• if the request 1 and 2 are issues by the same client, the top picture is what is really happening, and the bottom picture is not possible via HTTP which is request/response protocol. You would need to have a push or simulate it.
• if the requests are issued by different client then the top scenario would only happen if you had allowed only single thread/prefork on the balancer, which makes no sense. The bottom picture is what is happening in fact when 2 clients create 2 requests.

So I am still not sure what you want :-)

[minhld]

Hello Wolf-Dieter and Radoslav,

Thank you very much for the answers.

Try the test and increase the number of clients and you will see that you will reach a point where one JBoss instance is overloaded and you can not increase the throughput.

Yes, I made a test like below

• I built a JBoss cluster with 3 nodes, and they worked properly (I'm sure about this).
• I rebuilt the Apache httpd with mpm_worker module and use mod_jk as a load balancer. The configuration I set for the mpm_worker is default as in http://httpd.apache.org/docs/2.0/mod/worker.html
• I launch 10 clients simultaneously, with each sends 10000 requests to (test1) Apache server, and (test2) a node in the cluster.

But unfortunately the test1 is still slower than test2.

• if the requests are issued by different client then the top scenario would only happen if you had allowed only single thread/prefork on the balancer, which makes no sense. The bottom picture is what is happening in fact when 2 clients create 2 requests.

Yes this is what I expected Apache will do, but the result I got from the test with 10 simultaneous clients is not like this

I think from your two affirmations, I might be wrong in setting up Apache server somewhere (but honestly I didn't know where I went wrong since all configurations from Apache, mpm_worker and mod_jk are set by default according to the guidelines).

The suggestion using Apache along with mod_cluster is quite interesting and I will try this next week. Thank you and have a nice weekend!

[rhusar]

Definitely try mod_cluster,

site: http://www.jboss.org/mod_cluster/

docs: http://docs.jboss.org/mod_cluster/1.1.0/html_single/

Let us know how it went ;-)

Rado

[minhld]

Hi all,

Sorry for lately reply, these days I had some other works to do.

I have set up mod_cluster (with httpd) and test it with the load-demo application. This pictures below are what I got from the test. It is still not as good as I expected...

My cluster: 2 JBoss nodes, load balancer: mod_cluster (integrated with httpd),

The paragraph below is the configuration appending to httpd.conf (for mod_cluster)

LoadModule proxy_module modules/mod_proxy.so

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

LoadModule slotmem_module modules/mod_slotmem.so

LoadModule manager_module modules/mod_manager.so

LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

LoadModule advertise_module modules/mod_advertise.so

Listen 192.168.1.251:6666

<VirtualHost 192.168.1.251:6666>

  <Directory />

    Order deny,allow

    Deny from all

    Allow from 192.168.1.

  </Directory>

  KeepAliveTimeout 60

  MaxKeepAliveRequests 0

  ManagerBalancerName mycluster

  AdvertiseFrequency 5

</VirtualHost>

1. Test with 80 threads

With cluster:

Speed (add up 2 nodes): ~780 requests/s

With node #1, and node #2, separately:

Speed (each node): ~780 requests/s

2. Test with 1000 threads

With cluster

Speed (add up 2 nodes): ~ 2000-2500 requests/s

With node#1

Speed (node #1): ~9500 requests/s (Amazing!)

3. Test with 10000 threads

With cluster

Speed (2 nodes) ~2000-3000 requests/s. Please notice that more than 9100 threads are failed during the load.

With node #1

Speed ~10000 requests/s. Notice that number of failed threads is ~8500 (better than in cluster mode).

I'm really so confused about the results. Could you please check it your side and let me know you opinion? Thank you!

[rhusar]

So it look that cluster is 800 r/s combined and acting each node standalone its 1600 r/s combined?

For the failed requests you need to setup your Apache worker/prefork to accept more connections.

[minhld]

Hi Radoslav,

Please take a look at the test#2, where I start 1000 clients at a time (no failure reported).

Speed of cluster (2 nodes combined) is 2000-2500 r/s, whereas speed on node#1 is 9500 r/s.

For the failed requests you need to setup your Apache worker/prefork to accept more connections.

Yes, you are right, I also found the problem when I looked into the server log.

[minhld]

Hi all,

I increased the number of nodes in my JBoss cluster, I noticed the more nodes are added, the lower speed on each node in the cluster.

In particular: I tested with 1000 simutaneous clients

• Connect directly to a standalone JBoss server:
  • Speed ~ 9000-10000 requests/sec
• Connect to the JBoss cluster (through apache with mod_cluster - default configuration) with 2 nodes:
  • Speed on each node ~3000-5000 requests/sec.
• JBoss cluster with 3 nodes
  • Speed on each node ~2000-3000 requests/sec.
• JBoss cluster with 4 nodes
  • Speed on each node ~1500-2500 requests/sec.

So by this result, I may doubt about the JBoss cluster performance Do you have any idea for that?

[rhusar]

BTW Any updates on this?

Scaling:

if you are using TOTAL = FULL replication that means that every time you add a new node each web/ejb session will have to be replicated one more time. So yes, the more you add the slower each become. Thats why there is concept of BUDDY replication where each server elects only one (or more if you like) "buddy" where keeps the backup of someone elses session. So the fault tolerance is number of buddies - 1. Only with buddy replication you can scali linearly.

HTH,

Rado