-
1. Re: Jboss performance - not using all available resources
4l3xb Jun 11, 2007 11:16 AM (in response to jamesdonnelly256)does the queue build a backlog of messages when they're being add to the queue?
what os are you using and how did you determine only 1 cpu was being maxed?
could you take a thread dump?
rgds -
2. Re: Jboss performance - not using all available resources
peterj Jun 11, 2007 11:16 AM (in response to jamesdonnelly256)Operating system? 32 bit? 64 bit?
We run JBoss on multi-server configurations often, using SpecJAppServer benchmark as the test bed. But we also have some other hand-coded tests. One of the things you will find is that an app server will not scale beyond 4 CPUs. This applies to any app server, not just to JBoss. In fact, you should affinitize the app server to 2, 3 or 4 of your CPUs - the actual number depends on your application (try each one, pick the best), and preferrably CPUs that occupy a common bus (usually 4 CPUs share a bus).
Regarding Xmx/s, on a 32-bit system we have not been able to go over 1.7 GB of heap, on a 64-bit system only RAM is the limit (though with a really large heap you get really long pause times during a major collection.
Also see http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002Tuning for some heap tuning hints.
Specifically http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002TuningComponentsJVM
This wiki entry contains most of what we learned running on large-scale systems. -
3. Re: Jboss performance - not using all available resources
jamesdonnelly256 Jun 12, 2007 5:17 AM (in response to jamesdonnelly256)Thanks for the replies.
4l3xb,
Yes, the request handler posts messages to the JMS queue and the onMessage method picks them up when they appear. Hitting the server with a benchmark tools gets the queue to grow to thousands of messages very quickly. The queue is then picked off very slowly by the message driven bean.
There are two ways we've determined CPU usage. From sysstat, we've used:
# mpstat -P ALL
We've also used top, with the '1' command to list individual CPU usage.
PeterJ,
Sorry, forgot to mention.
We're running a 32bit Linux 2.6 kernel from Kubuntu.
We could try affinitizing the JVM to certain CPUs. Would you be able to advise how this can be accomplished? -
4. Re: Jboss performance - not using all available resources
jamesdonnelly256 Jun 12, 2007 10:13 AM (in response to jamesdonnelly256)4l3xb,
I've taken 2 thread dumps at different times. I don't know the best way to analyze these.
Should I post them here? They are a bit large.
Thanks. -
5. Re: Jboss performance - not using all available resources
peterj Jun 12, 2007 10:47 AM (in response to jamesdonnelly256)We could try affinitizing the JVM to certain CPUs. Would you be able to advise how this can be accomplished?
While Windows gives you tools to change affinity (Task Manager), Linux typically does not (I say typically because I am not familiar with the tools Ubuntu provides, I tend to use Fedora Core and occasionally RHEL). You usually have to write your own code to set affinity. Google the Portable Linux Processor Affinity API. -
6. Re: Jboss performance - not using all available resources
4l3xb Jun 13, 2007 1:54 AM (in response to jamesdonnelly256)I've taken 2 thread dumps at different times. I don't know the best way to analyze these.
Should I post them here? They are a bit large.
I wouldn't post them as you say they're probably large, it's probably no necessary to post the whole thing.
I'd start by looking for stack traces of threads that are processing your messages. search for jms/jboss packages/classes/methods or something in your code.
how many threads are there processing the msgs?