Infinite Garbage Collection Issue
moontz Jun 14, 2010 6:02 PMWe are getting some strange behavior on the system recently. This is JBoss 5.1 w/ JDK 6. Seems the garbage collector got stuck in some type of loop. Below is a snippet from the GC logs.
423612.480: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)] GC time would exceed GCTimeLimit of 98%
, 3.9623100 secs] [Times: user=7.42 sys=0.00, real=3.96 secs]
423616.447: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)] GC time would exceed GCTimeLimit of 98%
, 4.0101730 secs] [Times: user=7.54 sys=0.00, real=4.02 secs]
423620.458: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)] GC time is exceeding GCTimeLimit of 98%
, 4.0221410 secs] [Times: user=7.34 sys=0.00, real=4.02 secs]
423624.482: [Full GC [PSYoungGen: 233088K->233088K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)] GC time would exceed GCTimeLimit of 98%
, 4.1043720 secs] [Times: user=7.54 sys=0.00, real=4.11 secs]
423628.588: [Full GC [PSYoungGen: 233088K->233088K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)] GC time would exceed GCTimeLimit of 98%
This just continued for about 20 minutes before I shut the process down manually. Obviously this was sucking up all CPU and made the system unuasble. Has anyone seen this before? Any help would be much appreciated. Also, here is our run.conf:
JAVA_OPTS=" -server -Xms1g -Xmx1g -XX:PermSize=128m -XX:MaxPermSize=128m -XX:NewRatio=2 -Dorg.jboss.resolver.warning=true -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+DisableExplicitGC"
# Verbose Garbage Collection
JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:/opt/jboss/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
# Enable the jconsole agent remotely on port 12345
#JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=12345"
#JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
#JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
# JVM Debug Options
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/jboss/log -XX:ErrorFile=/opt/jboss/log/java_error%p.log"
Only thing I can think of is we start a sub process via ProcessBuilder class to make a call to lp (command line printing). I know they were testing that today among many other things so wondering if that threw things for a loop. When unit tested and system tested, works fine though.
Any thoughts?
Thanks,
Brian