Can you provide more information ?
Is the application measurably slow ? If so, how do you know ? If the CPU utlil. is low, perhaps the load you are applying is simply insufficient to keep the boxes busy.
If the app is slow, you need to figure out where the botleneck is. Last time I saw this particular symptom, it was a misconfigured NIC on the database server. This was resulting in very slow DB queries, but no visible contention on the app servers or the DB server.
Thanks for the response. Yes, we are seeing slowdown, measurably. The CPU goes to 100%, but it's not the expected way. We have 4 processors. Each processor gets up to 25% (never any higher): 100% = 25% X 4. Once this threshold is reached, JBoss crawls. This is observed in a production system for a highly public, well-known, and high-traffic application. This application is middle-ware business logic only (EJBs, POJOs, etc). There is no web server or JSP/servlet load on this box. And it's not very DB intensive.
So, the load is sufficient and we have ruled out the DB as the bottle-neck. We have never seen this application (or any other of our production apps running on JBoss) produce a CPU utilization higher than 25% per processor.
Any ideas? Has anyone ran into a similar situation? Does anyone have experience running JBoss on multi-CPU box running Linux 9.0? What do you think about the possibility that we may have our thread count too low, so that it could be maxing out our threads, but never fully utilizing our CPUs?
As always, any help is greatly appreciated.
How do you determine CPU utilization?
Perhaps it is scaled by the number of processors, so 25% already means maximum usage?
I might be guessing but do you have a hyperthreading CPU or do you actually have 4 cpu's. Could this not explain what you observe?
Just so we undestand what the CPU utilization is, can you post a sar -u or a top output ?
What king of JVM are you running ?
Aren't you running it in green thread mode ?
Thanks for the responses. I will try to answer all questions to help me get to the bottom of this.
By CPU Utilization, I mean we are never seeing all 4 CPUs go above a combined amount of 120%. To illustrate, here are a few "top" printouts:
CPU0 states: 15.1% user, 3.3% system, 0.0% nice, 81.1% idle
CPU1 states: 15.2% user, 2.4% system, 0.0% nice, 81.2% idle
CPU2 states: 6.0% user, 1.2% system, 0.0% nice, 92.1% idle
CPU3 states: 64.3% user, 7.2% system, 0.0% nice, 27.3% idle
CPU0 states: 7.3% user, 3.2% system, 0.0% nice, 89.0% idle
CPU1 states: 6.1% user, 1.3% system, 0.0% nice, 91.4% idle
CPU2 states: 1.0% user, 0.4% system, 0.0% nice, 98.0% idle
CPU3 states: 83.2% user, 3.4% system, 0.0% nice, 12.2% idle
CPU0 states: 6.4% user, 1.1% system, 0.0% nice, 91.3% idle
CPU1 states: 12.1% user, 1.4% system, 0.0% nice, 86.0% idle
CPU2 states: 3.3% user, 1.3% system, 0.0% nice, 94.2% idle
CPU3 states: 76.1% user, 3.3% system, 0.0% nice, 19.4% idle
We have 4 CPUs, not 1 Hyperthreaded CPU. We are using the standard Linux JDK 1.4.2 from Sun with the standard VM args in JBoss's run.sh, so as far as I know, we're not using Green Threads.
What I'm looking for is something similar to this "top" output (NOTE: not a real output, just something that would be ideal)
CPU0 states: 85.4% user, 6.3% system, 0.0% nice, 7.1% idle
CPU1 states: 94.4% user, 4.0% system, 0.0% nice, 0.5% idle
CPU2 states: 84.3% user, 7.3% system, 0.0% nice, 7.2% idle
CPU3 states: 86.2% user, 7.0% system, 0.0% nice, 6.1% idle
Is there anything in the JDK or JBoss that consciously prevents us from using all available CPU? Any other ideas or tuning suggestions or light shed is greatly appreciated.
Some operations in JBoss or your O/S may not be very parralelizable, blocking the other CPUs. There are tools to find out where your application (or JBoss) may be blocking the other CPUs, such as JProbe or JProfiler.