performance issues - inconsistent results for multiple concurrent users

[jainee]

We are deploying or web application on Jboss EAP 5.2. The web application receives a webservice call to process requests.

The two instances of production profile is deployed on two different boxes and request is distributed using a http server. The two boxes are 2 core machines.

We have tried to tune the jboss profile as described on various results on forum - thread group, memory settings, and other JAVA_OPT parameters.

We get very consistent results and bechmarked results while running the tests with 16 simultaneous users.

If we increase the number of users to 20 (10 for each box) - we get inconsistent results for multiple runs. We response time just doubles up and there is no consistent pattern on the same.

I believe if multiple users are not supported by application server or box level settings - we would get bad performance but atleast consistent results.

Is there any recommendations on the performance settings to get the best results when multiple users are invoking the webservice simultaneously?

Update - There is no CPU/threads starvation while running the tests.

[wdfink]

Hello,

welcome to the forums

First you should enabe the GC logging or check the GC behaviour with a external program like jstat the heap setting is mostly the primary problem for such problems.

[jainee]

Thank you for your response.

We have been keeping an eye on the GC, and we have used different GC policies to test the results. We didn't still see the consistent results on linux. We have configured more heap size 3G, which is definetely more than required in our case.

I would enable logging and check out the GC behavior in more detail, though we eliminated it as an option initially because we see only 60% of the heap getting utilized during our run. And we do not see any CPU utilization.

Somehow the threads are getting collected in the queue resulting in more time for processing the requests. It becames very incosistent for different runs. Any other pointers will be highly apprecaited.

[wdfink]

If you run 10 request with a single node did you have the same differences?

You need to check where the time is spend, i.e. is it possible to run 10 parallel request on each node. Where the time is spend/lost.

It is difficult to find such issues as there are many different reasons for that

[jainee]

We have tried invoking the single node directly and we have had even worst performance. The response time shooted upto 3 times than expected.

If we try to check the trace, the most of the time is spent during the xml functionality at different parts of the code. Its parsing, retrieving the attribute, or transformation, and its quite random during different invocations.