3 Replies Latest reply on Dec 1, 2008 11:10 PM by rhills

    JBoss Community DOC-10217 - advice on tomcat connector threa

    rhills

      I'm going through http://www.jboss.org/community/docs/DOC-10217 to help tune our jboss server for production.

      In the section headed "Tomcat", there is advice about tuning thread parameters for the http connector and I'm not sure if I'm understanding this right.

      Specifically, I'm looking at minSpareThread and maxSpareThread attributes.

      In DOC-10217 it says the following about minSpareThread:

      You should have minSpareThreads equal just a little more than your normal load

      and
      minSpareThreads means "on start up, always keep at least this many threads waiting idle"


      However, the second quote above doesn't seem to tell the whole story. In the Tomcat docs (http://tomcat.apache.org/tomcat-5.5-doc/config/http.html) it says the following about minSpareThreads:
      The number of request processing threads that will be created when this Connector is first started. The connector will also make sure it has the specified number of idle processing threads available. This attribute should be set to a value smaller than that set for maxThreads. The default value is 4.


      My interpretation of the Tomcat docs description is that this value should be set to a figure that equals the usual (or perhaps peak?) thread turnover. I'm not sure that is the same thing as "just a little more than your normal load", but maybe that is what the author of the JBoss "slimming" doc meant?

      In DOC-10217 it says the following about maxSpareThread:

      You should have maxSpareThreads equal just a little more than your peak load

      and
      maxSpareThreads means "if we ever go above minSpareThreads then always keep maxSpareThreads waiting idle"


      The Tomcat docs seem to describe maxSpareThreads very differently:

      The maximum number of unused request processing threads that will be allowed to exist until the thread pool starts stopping the unnecessary threads. The default value is 50.


      Now, I'm not sure exactly what the Tomcat Docs mean by "the unnecessary threads" that are being stopped, but my interpretation of this is that once the number of unused threads drops below this figure, the server starts hunting for other threads that are not yet marked as unused but might be dead or dying.

      If that interpretation is correct, then setting this figure to "just a little more than your peak load" as suggested in the JBoss slimming document seems to me that it would be too high. I'd have thought this figure would need to be set relatively low. I note that the ratios between the default values specified by the Tomcat docs (maxThreads=200, minSpareThreads=4, maxSpareThreads=50) don't seem to me to match the recommendations in the JBoss Slimming document.

      I expect I'm not understanding either document properly, but I'd appreciate some discussion on this to clarify things.


        • 1. Re: JBoss Community DOC-10217 - advice on tomcat connector t
          peterj

          The problem with using recommendations is that often they are specific to the environment of the user. And sometimes the recommendations are not necessarily the best especially if the recommendation is based on a performance test. For example, one of the recommendations is for setting sun.rmi.dgc.client.gcInterval to 60 minutes. Why 60 minutes? Because the specjappserver performance test runs in under 60 minutes so the "recommended" setting prevents stray full GCs from being called during the test run. But in a real life production environment, I would set it to once a day or once a week.

          Having said that, the Tomcat documents, since they describe the settings in general terms, are better references on how the settings work.

          (Examples below use the default values)

          MinSpareThreads is the minimum number of unused threads that will be kept available to handle new incoming requests. Example: if 20 threads are actively in use, there will be at least 24 threads (20 in use + 4 spare).

          A side effect is that this setting also dictates the initial number of threads created. Example: 0 threads in use, thus 4 threads. (I believe this is where the 'peak load' recommendation comes from - in a test such as specjappserver you can easily calculate the peak load and thus set minsparethreads to create that number of threads as part of system startup.)

          At some point the system might get busy and you end up with, say. 200 threads. Then everyone goes to lunch and only 20 threads are busy. When the idle thread timeout is reached, the thread count will be reduced to 70 (20 active plus 50 spare).

          Having said all of this, my testing has shown that Tomcat 6, and thus the versions of JBoss Web Server based on it, while it still accepts and stores these values no longer uses they to manage the threads. The min and max idle thread values are no longer checked in the source code (at least, it is no longer where it used to be checked and I haven't found any alternate locations where they are checked), and the observed behavior appears to validate the claim that they are no longer used.

          • 2. Re: JBoss Community DOC-10217 - advice on tomcat connector t
            rhills

            Hi Peter,

            "PeterJ" wrote:
            The problem with using recommendations is that often they are specific to the environment of the user. And sometimes the recommendations are not necessarily the best especially if the recommendation is based on a performance test. For example, one of the recommendations is for setting sun.rmi.dgc.client.gcInterval to 60 minutes. Why 60 minutes? Because the specjappserver performance test runs in under 60 minutes so the "recommended" setting prevents stray full GCs from being called during the test run. But in a real life production environment, I would set it to once a day or once a week.


            That's an excellent example. It's just a shame that the docs don't include that kind of qualification as well. I agree with you that it's almost impossible to provide recommendations that suit everybody. However I think that a set of recommendations for a specific set of circumstances with some explanation of why each value has been chosen would be a very useful document that most of us could then extrapolate to our own circumstances.

            "PeterJ" wrote:
            Having said that, the Tomcat documents, since they describe the settings in general terms, are better references on how the settings work.

            (Examples below use the default values)

            MinSpareThreads is the minimum number of unused threads that will be kept available to handle new incoming requests. Example: if 20 threads are actively in use, there will be at least 24 threads (20 in use + 4 spare).

            A side effect is that this setting also dictates the initial number of threads created. Example: 0 threads in use, thus 4 threads. (I believe this is where the 'peak load' recommendation comes from - in a test such as specjappserver you can easily calculate the peak load and thus set minsparethreads to create that number of threads as part of system startup.)

            At some point the system might get busy and you end up with, say. 200 threads. Then everyone goes to lunch and only 20 threads are busy. When the idle thread timeout is reached, the thread count will be reduced to 70 (20 active plus 50 spare).


            I agree, the Tomcat docs are better in this area, but I find your descriptions above even better still. It's a shame they're not in the Tomcat docs.

            "PeterJ" wrote:
            Having said all of this, my testing has shown that Tomcat 6, and thus the versions of JBoss Web Server based on it, while it still accepts and stores these values no longer uses they to manage the threads. The min and max idle thread values are no longer checked in the source code (at least, it is no longer where it used to be checked and I haven't found any alternate locations where they are checked), and the observed behavior appears to validate the claim that they are no longer used.


            Hmmm, for some silly reason, I'd thought JBoss was using 5.5 - I probably saw the number "5.5" somewhere and made the assumption without checking. Looking then at the http://tomcat.apache.org/tomcat-6.0-doc/config/http.html, no mention is made of the maxSpareThreads and minSpareThreads attributes, so that adds further evidence to your observations. Thanks for pointing that out.

            Presumably, Tomcat now tunes itself for these parameters?


            • 3. Re: JBoss Community DOC-10217 - advice on tomcat connector t
              rhills

               

              "rhills" wrote:
              Hmmm, for some silly reason, I'd thought JBoss was using 5.5


              Just to confirm what Peter said earlier, according to the release notes for JBoss 4.2.3.GA, Tomcat 5.5 was replaced by Tomcat 6 in JBoss 4.2.0.GA.

              See here: http://sourceforge.net/project/shownotes.php?release_id=614346&group_id=22866
              under the heading "JBossAS 4.2.0.GA"