2 Replies Latest reply on Jun 14, 2010 9:37 PM by moontz

    Infinite Garbage Collection Issue

    moontz

      We are getting some strange behavior on the system recently.  This is JBoss 5.1 w/ JDK 6.   Seems the garbage collector got stuck in some type of loop.  Below is a snippet from the GC logs.

       


      423612.480: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)]      GC time would exceed GCTimeLimit of 98%
      , 3.9623100 secs] [Times: user=7.42 sys=0.00, real=3.96 secs]
      423616.447: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)]      GC time would exceed GCTimeLimit of 98%
      , 4.0101730 secs] [Times: user=7.54 sys=0.00, real=4.02 secs]
      423620.458: [Full GC [PSYoungGen: 233087K->233087K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)]      GC time is exceeding GCTimeLimit of 98%
      , 4.0221410 secs] [Times: user=7.34 sys=0.00, real=4.02 secs]
      423624.482: [Full GC [PSYoungGen: 233088K->233088K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)]      GC time would exceed GCTimeLimit of 98%
      , 4.1043720 secs] [Times: user=7.54 sys=0.00, real=4.11 secs]
      423628.588: [Full GC [PSYoungGen: 233088K->233088K(466048K)] [ParOldGen: 1398142K->1398142K(1398144K)] 1631230K->1631230K(1864192K) [PSPermGen: 120708K->120708K(262144K)]      GC time would exceed GCTimeLimit of 98%

       

       

      This just continued for about 20 minutes before I shut the process down manually.  Obviously this was sucking up all CPU and made the system unuasble.  Has anyone seen this before?  Any help would be much appreciated.    Also, here is our run.conf:

       


      JAVA_OPTS=" -server -Xms1g -Xmx1g -XX:PermSize=128m -XX:MaxPermSize=128m -XX:NewRatio=2 -Dorg.jboss.resolver.warning=true -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+DisableExplicitGC"

       

      # Verbose Garbage Collection
      JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:/opt/jboss/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"


      # Enable the jconsole agent remotely on port 12345
      #JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=12345"
      #JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
      #JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"

       

      # JVM Debug Options
      JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/jboss/log -XX:ErrorFile=/opt/jboss/log/java_error%p.log"

       

      Only thing I can think of is we start a sub process via ProcessBuilder class to make a call to lp (command line printing).  I know they were testing that today among many other things so wondering if that threw things for a loop.   When unit tested and system tested, works fine though.

       

      Any thoughts?

       

      Thanks,

      Brian

        • 1. Re: Infinite Garbage Collection Issue
          peterj

          Your NewRatio shoiuld be 3 or higher. Setting it to 2 is just asking for doing way too many old generation collections.

           

          I would avoid -XX:+UseParallelOldGC like the plague. Every time I have used it my old generation collections took 4 to 10 times longer to complete. And I tried this with multiple applications (both running within JBoss AS and with plain old Java SE apps).

           

           

          Looks like you are collecting GC data (-verbose:gc). Have you analyzed that data?  For suggestions on how to gather GC data and analyze it, see these white papers:

          Java Garbage Collection Statistical Analysis 101

          Java Garbage Collection Performance Analysis 201

          at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

          • 2. Re: Infinite Garbage Collection Issue
            moontz

            Thanks Peter.  I'll do some reading tonight    Also, are there any known issues that cause the issues we are having with the GC just looping Full GC and not really collecting.  Possibly young to large and not enough room in old so just keeps looping?

             

            Brian