JBoss threads are not getting freed up during load testing
Hi.
We are currently facing a JBoss threads related issue while load-testing our application.
Our configuration/details are:
App Server: JBoss:
- version: 5.1.0.GA AS, Non clustered
- maxThreads: 800
- Xmx = Xmx = 1024m (Total System RAM is 2GB)
- Database Connection Pool: Min=20, Max=800
Database: MySQL:
- version: 5.0.40 Enterprise Server (GPL)
Our Application Consists of:
- an ear comprising:
- a web application having an MVC framework (webwork)
- several EJB applications as needed by web application
Description:
We observed that when we send 50 concurrent requests, test passes successfully and after test is completed, by observing graphs of currentThreadsBusy from JBoss's web-console, we could observer, currentThreadsBusy count goes to zero. The same behaviour is observed for concurrent 100, 200, requests. However when we send more than 200 concurrent requests, we could observe that pose test-completion, the currentThreadsBusy count goes to zero and then after some time it goes up showing some threads are busy which stays busy and do not get freed up (which is 50 in case of 200 concurrent threads). We observed same behavior for concurrent 400 threads (we found around 100 threads are busy and stays forever), And in this manner as we go on increasing concurrent requests, more number of threads stays as busy forever. I have attached an image showing below graphs taken for such several readings and supporting the issue.
When JBoss does not find enough threads as free to server incoming requests, we observer below exception on JBoss console. Is this a proper error (means indicating JBoss needs more threads and hence say increase maxThreads count) ?:
16:33:06,998 ERROR [JIoEndpoint] Error allocating socket processor
java.lang.ArrayIndexOutOfBoundsException: 202
at org.apache.tomcat.util.net.JIoEndpoint$WorkerStack.pop(JIoEndpoint.java:778)
at org.apache.tomcat.util.net.JIoEndpoint.createWorkerThread(JIoEndpoint.java:661)
at org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:702)
at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:737)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:313)
at java.lang.Thread.run(Thread.java:662)
We tried tweaking JVM params, tweaking maxThreads count, even tries adding minSpareThreads="10" and maxSpareThreads="500" ... but the behavior remained same.
Could anyone please guides us in what can be wrong? What can we do further to resolve this issue?
I thank you all in advance for any help on this issue.
Thanks & BR,
Rupesh
