JBoss 5.1 - CPU high, App stops responding

[n2jaz]

I'm still learning, and I'm trying to troubleshoot an isue.

I have a "off the shelf" application deployed on JBoss 5.1. I start the appplication and all is well; after a few days the CPU never drops below 50% and eventually the application stops responding. I've tried going to http://<hostname>:port/ , just hangs... I'm trying to figire out where the problem lies..is it java or JBoss?

Any thoughts as to how I can find root cause?

JBoss 5.1

Sun Java 1.6_17 x64

[peterj]

Sounds like the app has an infinite loop. Take several thread dump about 10 to 20 seconds appart and look for a thread whose stack trace has not changed (and which is not waiting on an object or input - you can ignore those threads). Once you identify the thread, you can identify the code that caused the loop. If any classes from the "off the shelf" app are in the stack trace, let the developers that wrote the app know about the issue so that they can fix it.

[wdfink]

It might be also possible that the GarbageCollector run into trouble (this happen to our app, also no gracefull JBoss shutdown possible).

Do you activate the GC logging for JBoss?

Add something like this to run.conf Java parameter:

'-verbose:gc -Xloggc:${LOGDIR}/gc_`date +%Y%m%d%H%M%S`.log -XX:+PrintGCDetails -XX:+PrintTenuringDistri

bution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps'

here you will see that 'Application time' is only a few millis and 'Time where application stopped' are seconds ....

for additional info look to http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html

if not active you might use 'jstat -gc PID 1000 1000' and check whether gc is called very often.

For infos about jstat lokk here http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html

hope it helps

[jimyliu]

Like Peter said, get several thread dump will help to identify the root cause. (jstack is the one I normally use).  Also you can use JBoss console to get the thread dump and find which thread uses more cpu.