Might be that you have an issue with GC.
Check a thread dump whether a GC thread will consume all CPU.
Switch the gc logging on (see the oracle java documentation for that)
Maybe it is simple and you have less memory configured for the JVM
Looked at heap dump and Permanent space is 99%. Also polling process shows a lot of time accounted for GC.
I guess you are right on GC issue. Now how do I fix it?
Tried to bump Xmx and xx:maxpermsize(512mb) but Permanent allocation is non resposive. Stays around 50mb.
I got this enviroment recently and not well familiar with sun jdk 1.5_15 issues.
We can throw more memory at it but not sure how.
Running the permgen at 99% is not a problem - the JVM will usually allocate just barely enough space for it because it is not volatile. So if you say it stays around 50MB, than a maxpermgen of 64MB should be more than enough. (Allocating more can actually work against you because it reduces the amount of memory that can be allocated to other areas of the JVM.)
If you suspect GC is the issue, you need to set one of the GC logging options, such as "-verbose:gc -Xloggc:gc.log", the data will appear in gc.log. Analyzing that data will tell you what is going on. To get some ideas on doing this, see my white papers at:
The 101 and 201 papers are handy, as well as the "Analyzing GC with R" (the R script calculates the % of time the process sent in GC).
I think you should start by creating ThreadDumps and monitoring GC. Monitoring gc is easy, just add following to your start script:
-XX:+DisableExplicitGC -verbose:GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log
check if you have multiple "Full" gcs which last long time after each other. (grep -e Full gc.log)
to create ThreadDumps you can send
kill -QUIT pid on linux
use jstack <pid> > filename
or use a tool like moskito (moskito.anotheria.net) which can provide you embedded thread dumps.
You should create 10 thread dumps in short distance, maybe 3 seconds inbeetween and check for threads that are doing something strange. If you have an infinite loop in your code, you will be able to find it this way.
Did thread dump. Nothing special in there.
In the end it was a virtualization issue. After restarting vmware host os everything got back to normal. Thanks