JBoss 4.0.3 sp1 - Dropping Servlet Parameters

[rheim]

Has anyone seen a problem similar to the following:

We use OpenSTA to generate varying degrees of load to test our application j2ee app using JBoss/Hibernate/MSSQL.

At low levels (75 users generating approx 5000 requests/minute) the tests complete without issue. However, when we try and ramp up the test to more modest levels (150 users generating approx 10,000 requests/minute) our test fails. Most (over 99%) of all the calls are completed just fine, however on about 25 random Servlet requests (all POSTS), the parameters are null.

Checking our test we noted that OpenSTA sent each of the incorrect requests with the correct parameters. Also, using a packet sniffer on the webserver machine, we verified that the webserver received all the packets containing these parameters. However the application fails to properly process these requests since it thinks the parameters weren't submitted. We have debug code in place which logs a message once a servlet begins processing the request and can show all the parameters it received. For these cases, no parameters are there.

I'm guessing this is a load related item but being a JBoss novice I'm not sure where to really look.

We thought the problem might be related to the configuration of the HTTP connector and have played around with the acceptCount (varied from 100 to 500) and bufferSize (from default 2048 to 102400) parameters, but these didn't seem to make much of a difference. Also, we tried setting the maxKeepAliveRequests to -1 with the thought this was the issue, but alas to no avail.

The machine we are running is averaging 44% CPU utilization during the 150 user test and we have plenty of memory available (1.4 GB heap - avg 500 MB free during the test run). Also, using the same hardware we are able to test the previous version of our product using BEA Weblogic Server 7.0 (SP5) up to around 250 of these virtual users. Based on this, we know the system should be able to handle more than load than it currently can.

I'd be more than happy to deliver greater detail about specific configuration settings, just let me know.

Thanks in advance for you help.

[canders6]

After doing some additional testing I've narrowed it down to a thread issue.

It seems that once the load generates enough concurrent threads to hit the maxthread value the problem begins to appear.

By upping the maxthread parameter for the 150 VU test (70 threads) I was able to get the problem to go away. I'm not trying to figure out how large to make the threads to get our 200 (currently testing 125 threads) and 250 VU tests to work. When we were testing this on BEA Weblogic Server 7.0 (SP5), we only needed 40 threads to handle our tests (150VU - 250 VU). I know that not all webservers behave the same, but so far I admit I'm a bit surprised by the amount of difference I'm seeing here with the thread behavior.

While upping the thread count seems to "fix" the issue, this still sounds like some sort of defect to me. I would imagine that if all the threads are busy that they should be queing up until available threads become available. Each of our virtual users simulate end user activity and thus all of the users aren't busy all the time. There are definite peaks and valleys to the load.

Anyways, as I continue the testing I'll post the results here as I have them.