Sounds like your app has some infinite loops in it. The best way to find such loops is to take multiple thread dumps a few seconds apart and then look for threads that are busy and appear to be in the same code. Then look at the stack trace for your code and examine that code to find out why it is in a loop.
You should have a look to the JVM Garbage collection.
see  for JDK5. Often the GC goes crazy if you reach a load threshold.
Look whether something special is around this time.
Also it might be a memory leak, the heap is mostly used and there is not enough space to create new objects. You might check the VM from time to time with 'jstat' it is included in the java installation since 1.5
Peter, could you please explain a bit how would i go about taking thread dumps? and i imagine i would need to do this when the 300% CPU usage occurs and not during normal 0% CPU load?
Peter, would using the 'jstack' command produce the same output as killing the process with the -QUIT argument (SIGNAL 3)
We don't have access to the STDOUT on the production servers.
Besides producing a java thread dump do you have any other suggestions on debugging this issue?
@Wolf-Dieter Fink: I''ll look onto this as well, but i'm interested in hearing all possible options to debug this issue that is affecting the normal operation of our application.
See the GC article, you can activate the GC.logging in production (I've done it long time) it is not a visibly impact and give hints about the memory ussage in such cases.
jstat can be used to have a quick view of the heap allocation and gc activity (jconsole and visualvm might not aceptable in production)
Sending a QUIT signal will not kill the process, only cause a thread dump. Don't you have stdout redirected to a file? If so, the thread dump will show up there. And yes, jstack will also get the thread dump.
The problem just presented itself again.
This time I was able to generate a jvm thread dump. Can anyone take a look at it and tell me if they see something strange? http://pastebin.com/Sfs2zLGS
Thank you very much in advance!