Jboss performance - not using all available resources

[jamesdonnelly256]

Hi,

We have a new server with 8 cpus and 12gb of ram. We can't get Jboss to exploit the CPU resources available, and we're not sure where the bottleneck is.

JBoss 4.0.5 resides on the main server, with MySql 5 running on a separate box, connected via a dedicated private gigabit ethernet. We're using Java 5.

Requests to the application get appended to a JMS queue running against the Mysql 5 server mentioned above.

Once the queue is populated, a message driven bean picks up from it to action the requests.

The CPU usage never goes beyond 40-50% of one CPU when Jboss is processing the queue.

The database server is practically idling during this time, as it has a raid 10 array of very fast disks and 4 CPU's. The network traffic between JBoss and Mysql is minimal, so that isn't the bottleneck either.

I have tested a small multi-threaded counting program, which proves that the JVM is more than capable of exploiting all 8 CPU's at 100%

We've adjusted most parameters we can think of to get JBoss to go faster, but nothing makes much of a difference, including connection pool settings, number of concurrent mdb's, thread pool settings.

It's using 2gb of Xms/Xmx (this is the maximum for one JVM right?)

If anyone has any pointers towards where to look for a bottleneck it would be very helpful.

[4l3xb]

does the queue build a backlog of messages when they're being add to the queue?

what os are you using and how did you determine only 1 cpu was being maxed?

could you take a thread dump?

rgds

[peterj]

Operating system? 32 bit? 64 bit?

We run JBoss on multi-server configurations often, using SpecJAppServer benchmark as the test bed. But we also have some other hand-coded tests. One of the things you will find is that an app server will not scale beyond 4 CPUs. This applies to any app server, not just to JBoss. In fact, you should affinitize the app server to 2, 3 or 4 of your CPUs - the actual number depends on your application (try each one, pick the best), and preferrably CPUs that occupy a common bus (usually 4 CPUs share a bus).

Regarding Xmx/s, on a 32-bit system we have not been able to go over 1.7 GB of heap, on a 64-bit system only RAM is the limit (though with a really large heap you get really long pause times during a major collection.

Also see http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002Tuning for some heap tuning hints.

Specifically http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002TuningComponentsJVM

This wiki entry contains most of what we learned running on large-scale systems.

[jamesdonnelly256]

Thanks for the replies.

4l3xb,

Yes, the request handler posts messages to the JMS queue and the onMessage method picks them up when they appear. Hitting the server with a benchmark tools gets the queue to grow to thousands of messages very quickly. The queue is then picked off very slowly by the message driven bean.

There are two ways we've determined CPU usage. From sysstat, we've used:

# mpstat -P ALL

We've also used top, with the '1' command to list individual CPU usage.

PeterJ,

Sorry, forgot to mention.

We're running a 32bit Linux 2.6 kernel from Kubuntu.

We could try affinitizing the JVM to certain CPUs. Would you be able to advise how this can be accomplished?

[jamesdonnelly256]

4l3xb,

I've taken 2 thread dumps at different times. I don't know the best way to analyze these.

Should I post them here? They are a bit large.

Thanks.

[peterj]

We could try affinitizing the JVM to certain CPUs. Would you be able to advise how this can be accomplished?

While Windows gives you tools to change affinity (Task Manager), Linux typically does not (I say typically because I am not familiar with the tools Ubuntu provides, I tend to use Fedora Core and occasionally RHEL). You usually have to write your own code to set affinity. Google the Portable Linux Processor Affinity API.

[4l3xb]

I've taken 2 thread dumps at different times. I don't know the best way to analyze these.

Should I post them here? They are a bit large.

I wouldn't post them as you say they're probably large, it's probably no necessary to post the whole thing.

I'd start by looking for stack traces of threads that are processing your messages. search for jms/jboss packages/classes/methods or something in your code.

how many threads are there processing the msgs?