10 Replies Latest reply on Mar 13, 2012 11:09 AM by claudio4j

    Http connector performance test AS7

    claudio4j

      Hi, I am doing some performance testing related to the http connector of jbossweb, and I see AS 7.1 is not good as EAP 5.1.2.

      Therefore I ask some help, as the settings may need some adjustments.

       

      Sees like I can perform 800.000 connections from ab to EAP 5.1.2, but cannot submit 150.000 to AS 7.1

       

      The EAP 5.1.2 configurations and results at http://pastebin.com/F729pygq

      The AS 7.1 configurations and results at http://pastebin.com/qhrQ1def

       

      The relevant pieces are

       

      1) EAP 5.1.2 http connector (server.xml)

       

      connectionTimeout="10000" redirectPort="8443" acceptCount="50"  maxThreads="2048"

       

      2) EAP 5.1.2 JVM configurations

       

      -XX:NewSize=768m -XX:MaxNewSize=768m -Xms2048m -Xss256k -Xmx2048m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC

       

      3) ab test to EAP 5.1.2

       

      ab  -n 800000  -c 100 'http://10.1.201.99:8080/images/logo.gif'

       

       

      4) EAP 5.1.2 results

       
      Concurrency Level:      100
      Time taken for tests:   130.762 seconds
      Complete requests:      800000
      Failed requests:        0
      Write errors:           0
      Total transferred:      1690524667 bytes
      HTML transferred:       1452107085 bytes
      Requests per second:    6118.00 [#/sec] (mean)
      

       

      1) AS 7.1 http connector (standalone-lab1.xml)

       

          <subsystem xmlns="urn:jboss:domain:threads:1.1">
              <thread-factory name="my_tf" group-name="group1" thread-name-pattern="my_tf_" priority="1"/>
              <bounded-queue-thread-pool name="http_queue" allow-core-timeout="true">
                  <core-threads count="100"/>
                  <queue-length count="50"/>
                  <max-threads count="2048"/>
                  <keepalive-time time="30" unit="seconds"/>
                  <thread-factory name="my_tf"/>
              </bounded-queue-thread-pool>

       

                                                                                                                                 

       

      2) AS 7.1 JVM configurations

       

      -XX:NewSize=768m -XX:MaxNewSize=768m -Xms2048m -Xss256k -Xmx2048m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC

       

      3) ab test to AS 7.1

       

      ab  -n 50000  -c 100 'http://10.1.201.99:8180/logo.gif'

       

      I copied logo.gif from EAP 5.1.2 to AS 7.1

       

      4) AS 7.1 results

       

      $ ab  -n 150000 -c 100 'http://10.1.201.99:8180/logo.gif' ; date
      This is ApacheBench, Version 2.3 <$Revision: 655654 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
      Licensed to The Apache Software Foundation, http://www.apache.org/
      
      Benchmarking 10.1.201.99 (be patient)
      Completed 15000 requests
      Completed 30000 requests
      Completed 45000 requests
      Completed 60000 requests
      Completed 75000 requests
      Completed 90000 requests
      Completed 105000 requests
      Completed 120000 requests
      Completed 135000 requests
      apr_poll: The timeout specified has expired (70007)
      Total of 149958 requests completed
      Tue Feb 28 18:17:02 BRT 2012
      
      

       

      OS configuration

      $ ulimit -a
      core file size          (blocks, -c) 0
      data seg size           (kbytes, -d) unlimited
      scheduling priority             (-e) 0
      file size               (blocks, -f) unlimited
      pending signals                 (-i) 47440
      max locked memory       (kbytes, -l) 64
      max memory size         (kbytes, -m) unlimited
      open files                      (-n) 65536
      pipe size            (512 bytes, -p) 8
      POSIX message queues     (bytes, -q) 819200
      real-time priority              (-r) 0
      stack size              (kbytes, -s) 10240
      cpu time               (seconds, -t) unlimited
      max user processes              (-u) 16384
      virtual memory          (kbytes, -v) unlimited
      file locks                      (-x) unlimited
       

       

      Observations related to the test.

       

      a) before each test run, must wait for 1 minute for the OS to finish all TIME_WAIT connections. I found out that for my computer, around 27.000 connections is the limit

       

      b) I choose not to use ab keepalive, to force the connector usage

       

      c) When running ab against AS 7.1 it quickly top the max-threads=2048 and the connections keeps around 28.000 then ab finish with timeout errors.

       

      I ask if there are other settings I must look at or suggestions on how to configure the system.

        • 1. Re: Http connector performance test AS7
          ctomc

          Hi,

           

          how does you web subsystem configuration looks like?

           

          do you use native connector?

           

          --

          tomaz

          • 2. Re: Http connector performance test AS7
            claudio4j

            Hi Tomaz, it is posted in the previous message as well at http://pastebin.com/qhrQ1def

             

            I do not use the native connector in EAP 5.1.2 and AS 7.1

            • 3. Re: Http connector performance test AS7
              ctomc

              hi,

               

              that is configuration for threading subsystem not web.

              AS7.1 by default uses native connector if you do not disable it. that is why i ask.

              and some pref tests show that it works faster if you disable it.

               

              --

              tomaz

              • 4. Re: Http connector performance test AS7
                claudio4j

                > that is configuration for threading subsystem not web.

                 

                Yes. I configured the web subsystem to use the "http_queue" executor. Forgot to post it.

                 

                <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="http_queue" max-connections="50000"/>

                 

                > AS7.1 by default uses native connector if you do not disable it. that is why i ask.

                > and some pref tests show that it works faster if you disable it.

                 

                I see it installed at modules/org/jboss/as/web/main/lib/linux-i686/

                How can I disable it ? I renamed the extension to .so.no

                • 5. Re: Http connector performance test AS7
                  claudio4j

                  I removed the native connectors, but it made no difference for the tests results.

                   

                  Can you try it at your AS 7.1 installation ?

                  • 6. Re: Http connector performance test AS7
                    jaikiran

                    We disabled native by default in AS 7.1.0.Final (check the native=false attribute on the web subsystem in the configuration file). So native won't be used unless you explicitly enable it.

                    • 7. Re: Http connector performance test AS7
                      jfclere

                      Could you try without the executor and with org.apache.tomcat.util.net.Constants.WAIT_FOR_THREAD=true?

                       

                      setting max-connections could help too.

                      • 8. Re: Http connector performance test AS7
                        claudio4j

                        The web subsystem configuration

                         

                            <subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">
                                <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" max-connections="1000000"/>
                                <virtual-server name="default-host" enable-welcome-root="true">
                                    <alias name="localhost"/>
                                    <alias name="example.com"/>
                                </virtual-server>
                            </subsystem>

                         

                         

                        > Could you try without the executor and with org.apache.tomcat.util.net.Constants.WAIT_FOR_THREAD=true?

                         

                        Removing the executor drastically improved the performance results.

                         

                        Now I am able to submit 3.000.000 ab requests.

                         

                        > setting max-connections could help too.

                         

                        Yes, I set it to 1000000

                         

                        I will continue to perform tests here and will publish the results here.

                         

                        Thanks.

                         

                        Claudio

                        • 9. Re: Http connector performance test AS7
                          davidwery

                          Hi Claudio,

                           

                          If I understand you right, removing the executor and using non native connector is the best configuration ? Can you confirm ?

                           

                          Thanks,


                          David

                          • 10. Re: Http connector performance test AS7
                            claudio4j

                            Hi, David. Yes, I confirm, as far as my test in my system. As the test is fairly easy to reproduce, I recommend you to perform in your environment.