If you have a constant 100% CPU usage there is a serious problem. Consider that means each of the CPUs is running a thread that is not blocked all the time. When you run in debug mode it lowers your performance by introducing blocked IO calls (I am assuming it's not using NIO) but it decreases your CPU usage. Increased CPU usage can mean better overall performance through more efficient IO but not if the thing is pegged at 100%.
Cupla questions. How many processors on your 880? How does the load testing tool determine the CPU usage ? since it is running on a client it must communicate with the server to get those stats and mayby what is being reported is suspect.
Thanks for the response.
The CPU is not at a constant 100% usage - I only used this as a term indicating that the box itself was at it's full capacity. Not only did we watch th e CPU but all the other OS load paramerters also indicated full capacity. It is a 4 CPU box and the "Load" value reported via rstatd was also very high (16-20) when debugging was turned off and we hit 30 "users".
Webload uses rstatd to determine cpu usage/load and we often manually monitor these (using top, mpstat, etc...) and what it reports is always consistent with what we see manually.
Another indication that the server is "working better" when the debug flag is turned on is that we can pass more of the load onto the database server which indicates that we are getting more transactions per minute. With debugging off, we could only load the DB server (12 CPU sun box) to about 30% cpu and a load value of 1 - 2 no matter how many "users" we had hitting the jboss server - obviously not heavily loaded. When we switch to debug on, we're able to get the database box up to about 75% cpu usage, with a load value of 8-10 which is a much heavier load.
You're comment about blocked IO is interesting. The application is a massive re-implementation of an old mainframe cobol system. When it was coded there was very little thought given to performance and if we run it through a profiler, it is obvious that it spends most of it's time doing IO. For example, a single request from a user can result in over 30 database reads and 10 database updates in about 3 different transactions as well as a JMS message. Maybe somehow the different IO causes the db IO to be more efficient?
With all of that IO going on even one CPU would be snoozing most of the time unless there are several threads polling something-or-other. Might be worth your while to pinpoint those bottlenecks and come back and revisit the wacky debug behavior later. If there was anyway to circumvent JMS and queue in the application that could also be very telling. Good luck with your efforts.