Problem with configuration for high troughput

[sejmajnejm]

Hello,

I am trying to configure a cluster of Jboss servers (7.1.1.Final) with mod_cluster 1.2.
I am able to achieve e.g. 600 http requests/second on a single jboss instance (requests sent directly to this instance) but with mod_cluster it never goes better than e.g. 200 req/sec but only for a minute. Then the performance degrades.
I have 4 instances of jboss in a cluster. RAM and CPU is no issue.

Sample of the domain.xml configuration

subsystem web (of the profile I am using which is full-ha):

<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="http-blocking" max-connections="20000"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" enable-lookups="false" socket-binding="ajp" executor="http-blocking" max-connections="20000"/>

subsystem threads, e.g.:

 <blocking-bounded-queue-thread-pool name="http-blocking" allow-core-timeout="true">
                    <core-threads count="4000"/>
                    <queue-length count="2"/>
                    <max-threads count="8000"/>
                    <keepalive-time time="10" unit="seconds"/>
                </blocking-bounded-queue-thread-pool>

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
                <mod-cluster-config advertise-socket="modcluster" proxy-list="172.18.6.160:7777" advertise-security-key="123456789qwerty">
                    <dynamic-load-provider>
                        <load-metric type="busyness"/>
                    </dynamic-load-provider>
                </mod-cluster-config>
</subsystem>

In the Apache httpd I am using worker.c module for handling connections, but it looks like those values have little effect on the cluster performance.

<IfModule worker.c>
StartServers         3
ServerLimit         1000
ThreadLimit         256
MinSpareThreads     75
MaxSpareThreads     250
ThreadsPerChild     256
MaxRequestsPerChild  0
</IfModule>

LBstatusRecalTime 1

<VirtualHost 172.18.6.160:7777>

  <Directory />
    Order deny,allow
    Deny from none
    Allow from all
  </Directory>

  <Location /mod_cluster-manager>
   SetHandler mod_cluster-manager
   Order deny,allow
   Deny from none
   Allow from all
  </Location>

  KeepAliveTimeout 15
  MaxKeepAliveRequests 0

  ManagerBalancerName other-server-group
  AdvertiseGroup 224.0.1.106:23364
  AdvertiseFrequency 5
  EnableMCPMReceive

  AdvertiseSecurityKey 123456789qwerty

</VirtualHost>

Question is how mod_cluster should be configured to get high throughput? Please, I need some advice :-)

Thanks

A.

[mbabacek]

Hi AJ, I don't find anything obviously wrong with your configuration at first sight, but for the very high max-connections="20000", if you are not running a real many-cores beast, it may actually make the performance worse. I do not have any decent performance-oriented domain configuration on my hand at the moment, nor I have any perf-setup running with advertise-security-key and busyness load metric as it is in your case. Anyhow, I can show you a config that has a pretty decent performance. In order to help you further though, I would need to know what exactly you are testing - e.g. is there session replication going on? I guess so. Does session stickiness work properly (no unnecessary digging session data up from other nodes)? What does your test-servlet (app) look like? What OS do you use?

<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="3000"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" max-connections="3000"/>

<mod-cluster-config advertise-socket="modcluster" sticky-session="true" sticky-session-force="false" sticky-session-remove="false" connector="ajp">
   <dynamic-load-provider decay="2" history="10">
      <load-metric weight="1" type="cpu"/>
   </dynamic-load-provider>
</mod-cluster-config>

    <mod-cluster-config advertise-socket="modcluster" advertise="false" proxy-list="perf08:8888" sticky-session="true" sticky-session-force="false" sticky-session-remove="false" connector="ajp">
        <dynamic-load-provider decay="2" history="10">
            <!-- Metric #0--><load-metric weight="1" type="cpu"/>
        </dynamic-load-provider>
    </mod-cluster-config>

Try just to trust the default

<subsystem xmlns="urn:jboss:domain:threads:1.1"/>

for an instance, so as not to introduce to many variables into the test.

Regarding httpd, how about something like:

MaxKeepAliveRequests 0
KeepAlive On
KeepAliveTimeout 300
MaxRequestsPerChild 0
#The maximum length of the queue of pending connections. Just playing around...
ListenBacklog 4096
ServerLimit 200
ThreadsPerChild 50
StartServers 20
MaxClients 5000

And try a test-run without busyness load metric (use e.g. the cpu+heap instead) and without an explicit advertise security key setting.

If you can share the test application you are experimenting with, or at least its particulars (<distributable>? Uses EJB? What it does?) I might be able to tune the setup in my environment.

[sejmajnejm]

thank you for those tips. It worked much better this time. I changed the load metric, removed security key, connector="ajp" in mod-cluster-config was not accepted on my version of the server. But those lines in the Apache configuration did the trick for testing:

MaxKeepAliveRequests 0
KeepAlive On
KeepAliveTimeout 300

This is a POC application that is exposing rest web services and does some simple EJB/JPA logic behind that, no session replication is needed. 20000 is the target max capacity, but I could reach over 1000 req/sec which is satisfactory for now and this PoC. It looks like this is all about httpd tuning this time.
However, sorry to say, I am unable to perform more tests right now as I am expected to 'return' blades tomorrow...

Thanks again,

A.

[mbabacek]

I am glad it helped.

Regarding:

A J wrote:

...connector="ajp" in mod-cluster-config was not accepted on my version of the server...

You need to have that connector configured, e.g.:

<subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" native="false">
    <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="3000"/>
    <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" max-connections="3000"/>

    <virtual-server name="default-host" enable-welcome-root="true">
        <alias name="localhost"/>
        <alias name="example.com"/>
    </virtual-server>
</subsystem>