JBoss-Hibernate Performance Test

[erwinp]

Hello,

i've been doing some performance/capacity tests on an application which is deployed under JBoss 4.0.2 and which uses the Hibernate version that comes with JBoss.

During the test HTTP requests arrive at a servlet which forwards them to a local stateless session EJB. This EJB will check the request parameters and create a row in the database using Hibernate. The created POJO is turned into XML which is returned to the client.

My server is a Dell Inspiron 1750 with two Intel Xeon 2.40GHz CPU's, 1GB RAM, Mandrake Linux 10.0, MySQL 5.0.16 and BEA JRockit 1.5.0_04. To create a significant load i use about 6 client PC's.
I have tried about all the tips that are given at the tuning and slimming page (
http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossASTuningSliming).

The result of the tests are that i can easily put a load of 180 requests per second on the server. When i try 190 request per second i get exceptions on my clients. Using the WIKI tuning tips (espescially -Xss128k) i succeeded to increase the maximum capacity to 220 requests per second but after that it's really finished.
The time needed to handle a request with this maximum load is a bit more than 100ms, measured on the server (in the servlet).

The exceptions i get at the client are mostly "java.net.ConnectionException: Connection refused: connect" (i use Java clients and HttpURLConnection to execute the HTTP requests).

Are my test results of about 200 requests per second normal? I myself am a bit disappointed but OK... Am i missing something, is there something i can do to really improve performance drastically (not just a few percentages)? What would the influence of "heavier" hardware be...

Any help would really be appreciated!

Tanks a lot,
Erwin

[smarlow]

I believe that the "Connection refused" error means that the http client threads are all in use (you might check your Tomcat status via http://hostname:8080/status.)

Is your client load test running continuously (no think time). If so, your driving a much heavier load than 220 real clients (if they don't look at your web pages for more than 5 seconds, they are probably trying to bring your server down). I like to use the Apache jmeter tool which lets you specify the url and think time (it can spawn multiple client threads also).

Scott

[erwinp]

Hello Scott,

for my last "successfull" test i had 2200 client threads running distributed over about 6 client PC's. Each client thread will sleep for 10 seconds (minus request time) between two requests, the think time.
All these clients together put an average load of 220 requests per seconds on the server.

I increased the maximum Tomcat threads from 250 to 500 to 1000. This always seems to help a little bit but not really very much.

Any ideas?

Thanx,
Erwin

[smarlow]

You may know this already (I think its mentioned on the JBossASTuningSliming page). If the requests are directly or indirectly writing to a log file, they will almost run in single threaded fashion.

To eliminate this possibility, I would check the "jboss/server/yourprofile/log" for any log files that are growing during the test.

[erwinp]

My application uses LOG4J for logging to file. The level is set high enough to prevent any logging during the test (only exceptions would be logged but there aren't any).

[smarlow]

You said that you "slimmed" down the application server (at least you said that you read the wiki about doing that).

Did you turn off "development" mode in your server/slim/jbossweb-tomcat50.sar/conf/web.xml?

If no, then do so and try your perf test again.

If yes, check the java memory situation gc. If the free java memory number is around the same, memory is probably okay. If free java memory decreases (again after the gc) over time, then you need to figure out why.

If memory is okay, you need to dive into what is going on within your application when running under load.

[erwinp]

It took me a while but i was able to perform tests that add up to 600 rows in the database per second. This is quite good in opposite to the 200 i was able to do before.
I changed my application to use JDOM for transforming the POJO's into XML that's returned to the clients. Before i was using the default JDK's JAXP/TRAX stuff.
I also made some significant changes to my test client application. I changed some settings on the client machines as explained user
http://grinder.sourceforge.net/faq.html#windows-address-in-use. Futhermore the clients are using buffers to read the response which is much faster (shame on me, i know).
That sounds good but of course there are still a few problems.

After starting JBoss i perform a test with 250 virtual users, each user performs 12 requests per minute, so an average load of 50 requests per second is placed on the server and this during 5 minutes. This test always fails. After about 4 minutes it is as if the server blocks for a while and connections start to time out while other requests have a response time of more than a minute.
When i repeat this test (without restarting JBoss) and during the following tests, with a higher number of users, all goes fine.
It is as if the server needs to warm up before it can handle a higher load. This happens every time after restarting JBoss.
The same happens with my test with 2300 virtual users. After this test i can continue testing until 3000 virtual users (about 600 requests per second) after which the limit seems to be reached.

Somebody any idea of what's happening?

[markricard]

Have you tried to front your JBoss with Apache and have Apache handle your static content? That gave us a 30% improvment in concurrent users.