3 Replies Latest reply on Nov 14, 2012 10:24 AM by Michal Karm Babacek

    Problem with configuration for high troughput

    Newbie

      Hello,

       

      I am trying to configure a cluster of Jboss servers (7.1.1.Final) with mod_cluster 1.2.
      I am able to achieve e.g. 600 http requests/second on a single jboss instance (requests sent directly to this instance) but with mod_cluster it never goes better than e.g. 200 req/sec but only for a minute. Then the performance degrades.
      I have 4 instances of jboss in a cluster. RAM and CPU is no issue.

       

      Sample of the domain.xml configuration

      subsystem web (of the profile I am using which is full-ha):

      <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="http-blocking" max-connections="20000"/>
      <connector name="ajp" protocol="AJP/1.3" scheme="http" enable-lookups="false" socket-binding="ajp" executor="http-blocking" max-connections="20000"/>
      

       

      subsystem threads, e.g.:

       <blocking-bounded-queue-thread-pool name="http-blocking" allow-core-timeout="true">
                          <core-threads count="4000"/>
                          <queue-length count="2"/>
                          <max-threads count="8000"/>
                          <keepalive-time time="10" unit="seconds"/>
                      </blocking-bounded-queue-thread-pool>
      

       

      <subsystem xmlns="urn:jboss:domain:modcluster:1.0">
                      <mod-cluster-config advertise-socket="modcluster" proxy-list="172.18.6.160:7777" advertise-security-key="123456789qwerty">
                          <dynamic-load-provider>
                              <load-metric type="busyness"/>
                          </dynamic-load-provider>
                      </mod-cluster-config>
      </subsystem>
      

       

      In the Apache httpd I am using worker.c module for handling connections, but it looks like those values have little effect on the cluster performance.

      <IfModule worker.c>
      StartServers         3
      ServerLimit         1000
      ThreadLimit         256
      MinSpareThreads     75
      MaxSpareThreads     250
      ThreadsPerChild     256
      MaxRequestsPerChild  0
      </IfModule>
      
      LBstatusRecalTime 1
      
      <VirtualHost 172.18.6.160:7777>
      
        <Directory />
          Order deny,allow
          Deny from none
          Allow from all
        </Directory>
      
        <Location /mod_cluster-manager>
         SetHandler mod_cluster-manager
         Order deny,allow
         Deny from none
         Allow from all
        </Location>
      
        KeepAliveTimeout 15
        MaxKeepAliveRequests 0
      
        ManagerBalancerName other-server-group
        AdvertiseGroup 224.0.1.106:23364
        AdvertiseFrequency 5
        EnableMCPMReceive
      
        AdvertiseSecurityKey 123456789qwerty
      
      </VirtualHost>
      

      Question is how mod_cluster should be configured to get high throughput? Please, I need some advice :-)


      Thanks

      A.

        • 1. Re: Problem with configuration for high troughput
          Michal Karm Babacek Apprentice

          Hi AJ, I don't find anything obviously wrong with your configuration at first sight, but for the very high max-connections="20000", if you are not running a real many-cores beast, it may actually make the performance worse. I do not have any decent performance-oriented domain configuration on my hand at the moment, nor I have any perf-setup running with advertise-security-key and busyness load metric as it is in your case. Anyhow, I can show you a config that has a pretty decent performance. In order to help you further though, I would need to know what exactly you are testing - e.g. is there session replication going on? I guess so. Does session stickiness work properly (no unnecessary digging session data up from other nodes)? What does your test-servlet (app) look like? What OS do you use?

           

          <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="3000"/>
          <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" max-connections="3000"/>
          

           

          <mod-cluster-config advertise-socket="modcluster" sticky-session="true" sticky-session-force="false" sticky-session-remove="false" connector="ajp">
             <dynamic-load-provider decay="2" history="10">
                <load-metric weight="1" type="cpu"/>
             </dynamic-load-provider>
          </mod-cluster-config>
          

           

           

          Try just to trust the default

          <subsystem xmlns="urn:jboss:domain:threads:1.1"/>
          

          for an instance, so as not to introduce to many variables into the test.

           

          Regarding httpd, how about something like:

          MaxKeepAliveRequests 0
          KeepAlive On
          KeepAliveTimeout 300
          MaxRequestsPerChild 0
          #The maximum length of the queue of pending connections. Just playing around...
          ListenBacklog 4096
          ServerLimit 200
          ThreadsPerChild 50
          StartServers 20
          MaxClients 5000
          

          And try a test-run without busyness load metric (use e.g. the cpu+heap instead) and without an explicit advertise security key setting.

           

          If you can share the test application you are experimenting with, or at least its particulars (<distributable>? Uses EJB? What it does?) I might be able to tune the setup in my environment.

          • 2. Re: Problem with configuration for high troughput
            Newbie

            thank you for those tips. It worked much better this time. I changed the load metric, removed security key, connector="ajp" in mod-cluster-config was not accepted on my version of the server. But those lines in the Apache configuration did the trick for testing:

             

            MaxKeepAliveRequests 0
            KeepAlive On
            KeepAliveTimeout 300

             

            This is a POC application that is exposing rest web services and does some simple EJB/JPA logic behind that, no session replication is needed. 20000 is the target max capacity, but I could reach over 1000 req/sec which is satisfactory for now and this PoC. It looks like this is all about httpd tuning this time.
            However, sorry to say, I am unable to perform more tests right now as I am expected to 'return' blades tomorrow...

             

            Thanks again,

            A.

            • 3. Re: Problem with configuration for high troughput
              Michal Karm Babacek Apprentice

              I am glad it helped.

              Regarding:

              A J wrote:

               

              ...connector="ajp" in mod-cluster-config was not accepted on my version of the server...

              You need to have that connector configured, e.g.:

              <subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" native="false">
                  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="3000"/>
                  <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" max-connections="3000"/>
              
                  <virtual-server name="default-host" enable-welcome-root="true">
                      <alias name="localhost"/>
                      <alias name="example.com"/>
                  </virtual-server>
              </subsystem>