welcome to the forums
First you should enabe the GC logging or check the GC behaviour with a external program like jstat the heap setting is mostly the primary problem for such problems.
Thank you for your response.
We have been keeping an eye on the GC, and we have used different GC policies to test the results. We didn't still see the consistent results on linux. We have configured more heap size 3G, which is definetely more than required in our case.
I would enable logging and check out the GC behavior in more detail, though we eliminated it as an option initially because we see only 60% of the heap getting utilized during our run. And we do not see any CPU utilization.
Somehow the threads are getting collected in the queue resulting in more time for processing the requests. It becames very incosistent for different runs. Any other pointers will be highly apprecaited.
If you run 10 request with a single node did you have the same differences?
You need to check where the time is spend, i.e. is it possible to run 10 parallel request on each node. Where the time is spend/lost.
It is difficult to find such issues as there are many different reasons for that
We have tried invoking the single node directly and we have had even worst performance. The response time shooted upto 3 times than expected.
If we try to check the trace, the most of the time is spent during the xml functionality at different parts of the code. Its parsing, retrieving the attribute, or transformation, and its quite random during different invocations.