JBoss always at 100% CPU but not doing anything!

[jsinglet1]

Hi All,

I am having a horrible problem with my JBoss deployment. I am using Jboss-4.0.4CR2 with JAVA_OPTS="-server -Xmx2256m -Xms2256m -Djava.awt.headless=true", the JDK is BEA JRockit 1.5.0_07.

The application is a EJB 3.0 application, with lots of servlets, and JSF stuff on the client side. All of the session beans are stateless.

After about 1 day of running the CPU hits 100% and stays there until I reboot JBoss, then the process repeats. I've been profiling the code locally, but am unable to find a place where the code would be hanging up. Also, tailing server.log (with debug logging on) confirms that nothing is happening.

Question: What in the world can I do to figure out what is hanging the processor? I've confirmed that garbage collection is not the source by logging the garbage collection during this process.

I am at a total loss for how to move forward, but needless to say I find it totally unacceptable.

Can someone point me in the right direction?

Thanks,
JLS

[peterj]

When the processor hits 100%, take a thread dump (actually, take several) and see which thread(s) are running. That should narrow down the list of culprits.

Thread dump info:
http://www.jboss.org/wiki/Wiki.jsp?page=StackTrace

[triathlon98]

As a quick guess, check your memory consumption. I would guess that your system is actually constantly collecting garbage.

If this is the case, check your EJB3 caching setting. The default configuration uses a HasMap implementation which is not for production use (according to the Hibernate docs) as it does release much (or anything) from the cache. This should be replaced by a JBossCache instance.

Hope this helps you,
Joachim

[jsinglet1]

Hi Joachim,

Actually, I don't have the caching service active. (I don't have a ejb3-caching-service.xml (or whatever it is)) in my deploy directory.

I am willing to buy that GC might be slamming the CPU, but the used ram is always around 500MB, and I've allocated 2.25 GB of ram.

Is there something that I'm missing? Perhaps the cache IS on even without the deployment file?

Thanks,
JLS

[jsinglet1]

Hi Peter,

I'm trying to get that thread dump now, actually. I've changed the line in my run.sh to read:

-Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
-classpath "$JBOSS_CLASSPATH" \
org.jboss.Main "$@" > /tmp/jboss.trace


All of the chatter is going to this file, however, when I do a kill -3 to the process, nothing happens and no thread dump is emmited. Any ideas?

Thanks,
JLS

[otaylor]

The method using JMX linked to at the top of the page is likely easier; note
the note further down the page of doing the dump via twiddle.sh from the
command line.

[peterj]

Taking a thread dump is a function of the JVM, not of JBoss. Perhaps JRockit is placing the threadump somewhere else, like the IBM JVM does. You would have the check the JRockit documentation for details. Or it might not even support a thread dump, in which case I suggest trying the Sun JVM just long enough to track down this problem.

[smarlow]

Try redirecting standard error to the file as well:

-Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
-classpath "$JBOSS_CLASSPATH" \
org.jboss.Main "$@" > /tmp/jboss.trace 2>&1

And check that you are sending "kill -3" to the correct java process.

[jsinglet1]

Hi Scott, Peter, and Taylor,

I switched back to the Sun JVM and redirected stderr, and it works now. Now for the fun part: waiting for everything to blow up.

I'll post more details as they come.

thanks so much,
JLS