7 Replies Latest reply on May 5, 2014 5:53 AM by wdfink

    java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3

    jimmy56

      Dear all,

       

      I have a question about error:

       

      <source>java.lang.OutOfMemoryError: GC overhead limit exceeded</source>

       

      Our application runs on JBoss AS 4.2.3, where are deployed also another services, such as Jbpm 3.2.2 and ESB 4.5.GA. We have several deploys on production and among other problems there started to happen this problem on one of the environments. It's HW configuration shall be enough (16GB Ram, 16 CPU cores). JVM options are:

       

      {quote}export JAVA_OPTS="-Xms1024m -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"{quote}

       

      It happens twice till this time. Once I found that 20 minutes before OOME was raised there started problems with Connection Pool:

      2010-11-03 17:31:59,111 WARN  [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Destroying connection that could not be successfully matched:

      {quote}2010-11-03 17:31:59,111 WARN   [org.jboss.resource.connectionmanager.JBossManagedConnectionPool]  Destroying connection that could not be successfully matched:{quote}

       

      The other incindent this problem didn't happened.

       

      Stacktrace of Error itself follows (same for both cases, but the source of cause differs):

       

      {quote}Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
          at java.util.Arrays.copyOf(Arrays.java:2882)
          at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
          at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
          at java.lang.StringBuffer.append(StringBuffer.java:224)
          at org.hibernate.pretty.Formatter.out(Formatter.java:257)
          at org.hibernate.pretty.Formatter.misc(Formatter.java:216)
          at org.hibernate.pretty.Formatter.format(Formatter.java:178)
          at org.hibernate.jdbc.AbstractBatcher.format(AbstractBatcher.java:433)
          at org.hibernate.jdbc.AbstractBatcher.log(AbstractBatcher.java:424)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:505)
          at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:446)
          at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:140)
          at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
          at org.hibernate.loader.Loader.doQuery(Loader.java:673)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
          at org.hibernate.loader.Loader.loadCollection(Loader.java:1994)
          at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:36)
          at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:565)
          at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:60)
          at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1716)
          at org.hibernate.collection.AbstractPersistentCollection.forceInitialization(AbstractPersistentCollection.java:454)
          at org.hibernate.engine.StatefulPersistenceContext.initializeNonLazyCollections(StatefulPersistenceContext.java:844)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:241)
          at org.hibernate.loader.Loader.doList(Loader.java:2220)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
          at org.hibernate.loader.Loader.list(Loader.java:2099)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
          at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:65)

      {quote}

       

      Any ideas, how to fix or what is the cause of the problem?

       

      Thank you for any commets.

       

      Jakub.

        • 1. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
          f_marchioni

          Hello,

          about the garbage collector issue it seems you are spending too much in garbage collection with little memory recovery.

          Try using the -XX:-UseGCOverheadLimit switch and  check out this

          About the issue with the connection pool check out this thread: http://community.jboss.org/message/229436

          Hope it helps,

          Francesco

          jBoss tutorials

          • 2. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
            wdfink

            What happen to me is that the GC problem is caused by longer running requests.

            E.g. if your services running longer due to parallel load.

            - each service takes more time

            - each service blocks longer a part of heap

            - objects live longer than the GC will keep in survivor space or the survivor space is too small

            - the old generation becomes full with such short living objects

             

            If you have a problem like waiting for free (DB) connections you should solve this first, maybe the GC problem disapear.

            If not you have to analyze the memory consumption of young generation and increase the heap or set a different generation allocation.

            see http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html

            • 3. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
              peterj

              Setting -XX:-UseGCOverheadLimit will just turn off the symptom but not cure the problem. Most likely you have too many objects that are not being released (Wolf-Dieter's suggestion to track down the connection issue is a good one because connections can take up considerable heap space).

               

              You should also take some heap dumps and analyze them. You can take a heap dump using jmap or by setting -XX:+HeapDumpOnOutOfMemoryError. I recommend using VirtualVM (it comes with JDK 6 or you can downlaod separately; the JDK 6 one will handle JDK 5 heap dump). When running VirtualVM to analyze the dump you need to set the max heap size to about 3 times your dumped heap size (for you, that would be 6GB).

               

              You should also monitor the GC activity.  For suggestions on how to gather GC data and analyze it, see these white papers:

              Java Garbage Collection Statistical Analysis 101

              Java Garbage Collection Performance Analysis 201

              at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

              • 4. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
                f_marchioni

                I beg your pardon: -XX:-UseGCOverheadLimit will be just for keeping your server in production alive. You have definitely to discover what's the at the root of the problem. Supsecting too that it originates from a flawed database design-

                • 5. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
                  jimmy56

                  Thank you for your comments!

                   

                  I wondered if is it safe to just use the -XX:-UseGCOverheadLimit switch, but you are right that we should try it at least till we found what causes the problem (which process, ...). So I understand it is quite safe to use it (of course we will test it on test environment before putting on production)?

                   

                  Anyway I started to understand it is quite complex to find the problem - but we have to study little more the GC issues from referenced articles. I let you know, if we will find anything.

                   

                  Strange is that it happened till now only on one environment of around 20 all over.

                  • 6. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
                    sbharate

                    0 down vote 

                    I am working with JBoss application server and I am facing an issue related to

                    " Execution of JMS message listener failed java.lang.OutOfMemoryError: GC overhead limit exceeded "

                    I have set JAVA_OPTS from 0.5 GB to 1 GB in run.bat file. Also used an approach of taking Heap Dumps when this exception has occurred and analysed it using Eclipse Memory Analyser Tool (MAT), but I got only size of 20 MB Heap Dump.

                    Please suggest any suitable approach to get root cause or any resolution appraoch to fix this OutOfMemory : GC limits exceeds issue.

                    And even though this kind of exceptions are coming, System doesn't get creashed but the processing still in progress. i.e. No fatal exists

                    Many Thanks in Advance.

                    • 7. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded on JBoss AS 4.2.3
                      wdfink

                      Hello Swapnil,

                      you should not reactivate such old thread as the JVM and server are not the same.

                      You might have a look whether the jstat command can help to see whether it is a leak.

                      If you still have problems you should open a new discussion and provide the details