Servlet Threadpool increase

[raghu_c]

Hi,

We have this Jboss-Jetty Server hosting our webapp.
we use JBoss 3.2.0 that comes bundled with jetty

I am unable to configure the servlet thread pool to handle more requests simultaneously - while load testing.
I am able to increase the number of listeners on the http port (8080)
(in jbossweb-jetty.sar/META-INF/jboss-service.xml)
but these requests are not processed simultaneously since
an equal number of handlers are not present on the server.

Any pointers to configuring the same will be greatly appreciated.

Thanx and Regards,
Raghu

[jonlee]

You need to provide some information about your configuration for Jetty and also the manner in which you are testing. The max and min threads should not be set to the same value as last time I looked, the code starts taking terminating action on persistent connections if the number of idle threads is less than the minimum number of threads (min threads is the minimum number of idle threads) and you have reached the maximum number of threads.

Also no single processor system can handle true simultaneous servicing of requests. The number of requests you can handle at the same time is equal to the number of processors - obviously, so if you are generating requests as an impulse function of large value, your server is going to be spending a lot of time preparing resources to service the incoming requests as well as splitting CPU cycles between requests.

The larger the impulse value, the larger the response time will be as the time slicing between requests increases the elapsed time for individual requests to be processed.

So:
1) Do you ramp up your requests?
2) What is your initial process/thread count?
3) What happens?
4) Are any requests not serviced?
5) What symptoms or system artifacts do you receive to indicate that the system is not handling the load?
6) What is the setting you have for your listener?

This will give people here a better idea of what is happening.

[raghu_c]

Hi Jonlee -

I am on a 8 CPU i686 box with 4G ram.
I use java -Xms256m -Xmx 1536m

1) Do you ramp up your requests?
No we start with a preset 500 impulse

2) What is your initial process/thread count?
Viewing from the jmx console gives us a thread count of 10 for my application

3) What happens?
After a few requests are serviced , leaving all system parameters as defaults, the requests take an inordinately large time to return.

4) Are any requests not serviced?
Not many are left unserviced - but they take a long time

5) What symptoms or system artifacts do you receive to indicate that the system is not handling the load?
The CPU Load is 100 %
I got a java.lang.OutOfMemoryError from Jboss as it could not open a handle to a zip file.
So I increased the max-file descriptors to a million
( ulimit -n )
Then I got a jvm error asking me to increase stack size
I increased -Xss to 1024kb

which resulted in an OutOfMemoryError -
Cannot create native thread.

6) What is the setting you have for your listener?
min threads - 10
max threads - 2000


--- Is this got anything to do with the max threads / processes a user can handle?
If so how can increase that for RedHat linux 8.0?

Thanx ,
Raghu

[jonlee]

OK. You seem to have plenty of hardware. I'm still concerned with the initial hit. I'm not sure that jmx-console is handling things that well as the system is doing plenty of other things.

Try two things.

Generate your load and after completion, check the number of processes associated with the main Java thread - pstree.

Next try ramping up your load to see if it is able to handle things - I would suggest getting Apache JMeter, and perhaps spread the input across several client machines.

Also, tune your system and see if and when it can handle that impulse load. You can then try tuning your application deployment so that sufficient resources are started or available when your system is in a quiescent state - e.g. Set min numbers of beans in the pool and so on.

I suspect what we are seeing is that your system is spending a lot of time bootstrapping (creating beans, compiling or creating new servlet/JSP images and so on), which is delaying the system from actually performing the business logic. You may also want to create more standby servlet services - look at the webdefault.xml for Jetty as well as put more of your own servlets at a standby state (inital number set by initparam).

I don't think you are hitting the process wall yet and it is not something you want to consider as normally, you need to recompile the glibc library to increase the thread limit, etc.

The problem with the big bang theory of load testing is that it is chaotic and it is very difficult to interpret what is breaking. Ramping things up gives you time to consider internal effects of your load. Then you can look at the impulse handling.

With your expected load, you may also want to consider non-blocking IO for your HTTP listener. Jetty offers this but you will need to compile in the ChannelSocketListener.

Hope that helps.

[raghu_c]

Hi jonlee.

ya , i shall try to profile the system with gradually ramped up loads.
However, how can i increase the default pool size for my web apps?
correct me if i am wrong - jboss holds jetty as a service and deploys it into an "abstractwebcontainer" whatever that is.
i am unable to find any configuration setting for any such container into which jetty may have been deployed or any container of jetty's that holds my webapp

??
Raghu

[jonlee]

Web apps run in a slightly different manner to EJBs. Jetty or Tomcat are the web app containers. There are general controls for external incoming connections such as listeners. You also have default configurations for web apps - such as recognised MIME types - in defaultweb.xml or web.xml depending on whether you are using Jetty or Tomcat.

You can control the initial number of JSP servlets available to handle JSP requests (setting the load-on-startup value) and the initial number of default servlets to handle static content.

Within your own webapps, you can control the number of initial JSPs and servlets read on startup to service requests. This is like managing pools of EJBs. In a sense, you can manage your pools of servlets.

So you don't have pools of web applications, but pools of available service resources in the web application that you can control as well as the number of external connections to your container.

Hope that helps.

[jonlee]

I should point out that the thread pool that most people observe in jmx-console, is that used by the mini-webserver in JBoss. This is not to be confused with the Tomcat and Jetty servlet container listeners. The purpose of the mini-webserver is to simplify dynamic class-loading in RMI. It has a pool of 10 threads for this purpose.