Are you able to take a thread dump, and if so what does it show? Note that you might have to take multiple dumps to pinpoint the problem.
Last time I looked into a problem similar to this it turned out that with one of the deployed applications, if you did a specific sequence of requests, it would go into an infinite loop. It took getting only 3 requests in that state to get the processor up to 99% busy.
Here is the information from jmap and jstack for a server with CPU currently maxed at 99.9% as I described.
It is normal for many threads to be blocked, that just means they are not using the cpu at this time.
I am somewhat concerned with the "(Interpreted frame)" text on each method in the stack trace for the threads. I think that might mean that you are running an interpreter, instead of using the just-in-time compiler to get compiled code. I would have to research this more to be sure. What is the command line used to start the app server? Looks like your are giving the JVM free reign to resize the heap as necessary. I usually prefer setting specific heap sizes with min=max; usually heap of 1200MB and newsize of 300MB is a good starting point for tuning.
Other than that, I don't see anything that jumps out at me. Looks like most threads are waiting for something to do (you must be using Apache as a front end, considering the number of AjpProcessor entries; and it looks like some Oracle calls are in progress). You should take multiple thread dumps and compare the threads from each dump.
JAVA_OPTS="-server -Xms512m -Xmx2000m"
One thing that concerns me but may not be the problem is that I think this is a 64bit linux system, but java won't run with -d64.
# uname -a
Linux MTCAPPS5 2.6.5-7.286-smp #1 SMP Thu May 31 10:12:58 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux
# /usr/java/jdk1.5.0_11/bin/java -server -d64 -version
Running a 64-bit JVM is not supported on this platform.
It would appear that you installed the 32-bit JVM, not the 64-bit.
I also verified that your JVM is interprting the Jvaa byte code and not compiling it.
Where did you get this JVM?
Sun web site. You are right I downloaded the 32bit JVM instead of the 64bit. I now have the sun jdk-1_5_0_12-linux-amd64 JVM installed for JBoss. We'll see if the sun jdk 5.0 update 12, 64bit JVM fixes this problem.