-
1. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
peterj Feb 1, 2012 12:32 PM (in response to mcbahamonde)Sounds like your app has some infinite loops in it. The best way to find such loops is to take multiple thread dumps a few seconds apart and then look for threads that are busy and appear to be in the same code. Then look at the stack trace for your code and examine that code to find out why it is in a loop.
-
2. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
wdfink Feb 1, 2012 12:33 PM (in response to mcbahamonde)You should have a look to the JVM Garbage collection.
see [1] for JDK5. Often the GC goes crazy if you reach a load threshold.
Look whether something special is around this time.
Also it might be a memory leak, the heap is mostly used and there is not enough space to create new objects. You might check the VM from time to time with 'jstat' it is included in the java installation since 1.5
-
3. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
mcbahamonde Feb 1, 2012 12:40 PM (in response to peterj)Peter, could you please explain a bit how would i go about taking thread dumps? and i imagine i would need to do this when the 300% CPU usage occurs and not during normal 0% CPU load?
-
4. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
peterj Feb 1, 2012 12:41 PM (in response to mcbahamonde)See if this helps:
http://www.crazysquirrel.com/computing/java/basics/java-thread-dump.jspx
And yes your results will be much better if you do this then you hit 300% CPU usage! ;-)
-
5. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
mcbahamonde Feb 1, 2012 12:58 PM (in response to peterj)Peter, would using the 'jstack' command produce the same output as killing the process with the -QUIT argument (SIGNAL 3)
We don't have access to the STDOUT on the production servers.
Besides producing a java thread dump do you have any other suggestions on debugging this issue?
@Wolf-Dieter Fink: I''ll look onto this as well, but i'm interested in hearing all possible options to debug this issue that is affecting the normal operation of our application.
-
6. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
wdfink Feb 1, 2012 1:04 PM (in response to mcbahamonde)See the GC article, you can activate the GC.logging in production (I've done it long time) it is not a visibly impact and give hints about the memory ussage in such cases.
jstat can be used to have a quick view of the heap allocation and gc activity (jconsole and visualvm might not aceptable in production)
-
7. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
peterj Feb 1, 2012 1:07 PM (in response to mcbahamonde)Sending a QUIT signal will not kill the process, only cause a thread dump. Don't you have stdout redirected to a file? If so, the thread dump will show up there. And yes, jstack will also get the thread dump.
-
8. Re: JBoss [Trinity] 4.2.2.GA 100% CPU Usage
mcbahamonde Feb 6, 2012 2:34 PM (in response to peterj)The problem just presented itself again.
This time I was able to generate a jvm thread dump. Can anyone take a look at it and tell me if they see something strange? http://pastebin.com/Sfs2zLGS
Thank you very much in advance!