7 Replies Latest reply on Oct 15, 2015 7:59 AM by shawkins

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    tawalisa
    tawalisa Mar 16, 2011 1:40 AM

      Hi all,

       

                We meet a urgency question. When we do endurance test, Teiid server throw this exeception: java.lang.OutOfMemoryError: GC overhead limit exceeded. In this case, we do many deploy and undeloy VDB. And we do many "select " SQL command to Teiid.  After a few time, Teiid server will throw this exeception. I analysed dump file of teiid server. We can see. XAManagedConnection used 49.59% memory.  Our VDB has 2 datasource . Those datasource are XA. One of both connect oracle. Another datasource connect another VDB. This VDB has 2 datasoure ,too. They are XA. So from my architecture, if clieet use one connection to connect VDB, our VDB will use 4 XA Datasouce as lease. In memory of Teiid, we can find 169  org.jboss.resource.adapter.jdbc.xa.XAManagedConnection. They used 817m memory. Our teiid server java arguments is below.

      -Xms1303m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError

       

      I want to know whether this result is ok or memory of teiid has been leak?

       

       

       

      Biggest Top-Level Dominator Classes

       

      LabelNumber Of ObjectsUsed Heap SizeRetained Heap SizeRetained Heap, %
      org.jboss.resource.adapter.jdbc.xa.XAManagedConnection16921,632817,211,12849.59%
      org.jboss.mx.server.MBeanServerImpl296175,107,83210.63%
      org.teiid.metadata.Schema1008,800113,696,2646.90%
      oracle.jdbc.driver.T4CConnection56,88099,099,1206.01%
      org.jboss.virtual.plugins.context.zip.ZipEntryContext43752,44088,327,7285.36%
      char[]32,34863,251,65663,251,6563.84%
      org.jboss.classloader.spi.base.BaseClassLoader20133,76834,585,2002.10%
      java.util.jar.JarFile76160,88025,504,1041.55%
      org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy20128,94419,894,7521.21%
      org.jboss.profileservice.management.ManagementViewImpl122417,445,3281.06%

       

       

      Problem Suspect 1

       

      One instance of "org.jboss.mx.server.MBeanServerImpl" loaded by "org.jboss.classloader.spi.base.BaseClassLoader @ 0x2aaace604b50" occupies 174,974,024 (10.62%) bytes. The memory is accumulated in one instance of "EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry[]" loaded by "org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x2aaace603d38".

       

      Keywords
      org.jboss.classloader.spi.base.BaseClassLoader @ 0x2aaace604b50
      org.jboss.mx.server.MBeanServerImpl
      org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x2aaace603d38
      EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry[]


      Problem Suspect 2

       

      169 instances of "org.jboss.resource.adapter.jdbc.xa.XAManagedConnection", loaded by "org.jboss.classloader.spi.base.BaseClassLoader @ 0x2aaae105f960" occupy 817,211,128 (49.59%) bytes.

      Biggest instances:

      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf910cd78 - 60,778,320 (3.69%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf9137bf0 - 60,778,312 (3.69%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf910f230 - 60,778,296 (3.69%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf913cd68 - 60,778,008 (3.69%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf6900968 - 60,647,544 (3.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf6a49de0 - 60,647,544 (3.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf69a2018 - 60,646,592 (3.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf79ad400 - 60,642,520 (3.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf913c300 - 44,263,208 (2.69%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf75d10e8 - 44,132,304 (2.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf66adc10 - 27,617,784 (1.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf6629d88 - 27,617,496 (1.68%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf8caadf0 - 26,699,288 (1.62%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf72d48f8 - 26,565,472 (1.61%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aaaf5f9ddf8 - 26,565,464 (1.61%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aab2bf181b8 - 17,487,664 (1.06%) bytes.
      • org.jboss.resource.adapter.jdbc.xa.XAManagedConnection @ 0x2aab31b55418 - 17,487,392 (1.06%) bytes.

       

      Keywords

      org.jboss.resource.adapter.jdbc.xa.XAManagedConnection

      org.jboss.classloader.spi.base.BaseClassLoader @ 0x2aaae105f960

        • 1. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
          meltedmetal
          meltedmetal Mar 16, 2011 2:16 AM (in response to tawalisa)

          Hi，All

           

          I am in the same project with allen.

          More detail info of a XAManagedConnection.

          detailOfXAmanager.PNG

          From the above image, we could found several big char array in the instance of oracle.jdbc.driver.BufferCache.

          Whether is the data gotten from oracle database too large?

           

          Thanks.

           

          BRs,

          S.Q.R

            • Actions
          • 2. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
            rareddy
            rareddy Mar 16, 2011 10:29 AM (in response to meltedmetal)

            Allen,

             

            We have tested Teiid before for memory leaks and any issues we found were fixed. This particular error showing that it is happening in the Oracle driver. It is using the soft references, however they should be cleared if the VM is constrained for memory, and VM will clean those before a OOM. For testing sake

             

            1) See if there is new updated Oracle driver available.

            2) In -ds.xml file Oracle can you configure such that, you close the connection after let say after a minute of inactivity. Closing the connection should flush these immediately. (this is just to check)

             

            Thanks

             

            Ramesh..

              • Actions
            • 3. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
              shawkins
              shawkins Mar 16, 2011 12:32 PM (in response to rareddy)

              Allen,

               

              Too many SoftReferences can indeed cause problems if their clearing schedule is not proactive enough, see oracle java bug id 6912889.

               

              What VM are you using?

               

              If it's a oracle vm, you can follow the bug workaround by setting -XX:SoftRefLRUPolicyMSPerMB to a low value, even just 1 to see if your issue is resolved.

               

              The next question would be why the driver is creating such large buffers.  I don't see any information from oracle on that, but it could be related to the fetchSize.  Teiid is automatically setting the fetch size based upon the connector batch size.  You could also reduce the connectorBatchSize in the teiid-jboss-beans.xml and see if that helps.

               

              Steve

                • Actions
              • 4. java.lang.OutOfMemoryError: GC overhead limit exceeded
                shawkins
                shawkins Mar 23, 2011 2:43 PM (in response to shawkins)

                Allen,

                 

                Did you have any luck at resolving this?  Also if you use the latest 7.4 beta, you should see the memory size held by Schema objects significantly reduced.

                 

                Steve

                  • Actions
                • 5. java.lang.OutOfMemoryError: GC overhead limit exceeded
                  tawalisa
                  tawalisa Mar 24, 2011 11:10 PM (in response to shawkins)

                  Hello Steve,

                   

                            I have used 7.4 beta, And I am testing it now. By now, it looks better than 7.2

                   

                   

                  Allen

                    • Actions
                  • 6. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
                    jietao
                    jietao Oct 15, 2015 5:36 AM (in response to shawkins)

                    Hi Steve, I am using SQL Scrapbook (from teiid designer) to test my VDB to our MySQL database. I have similar problem:

                     

                    select * from tracking_subid_dimension

                     

                     

                    GC overhead limit exceeded

                     

                    shall I change some configuration files?

                     

                    Thx

                      • Actions
                    • 7. Re: java.lang.OutOfMemoryError: GC overhead limit exceeded
                      shawkins
                      shawkins Oct 15, 2015 7:59 AM (in response to jietao)

                      > shall I change some configuration files?

                       

                      Not until there's a determination of what is going on.

                       

                      Since this is an old thread, you should start a new one.  Ideally you could include details from a heap dump showing how the heap is being utilized.  Also include the query plan of the query you are performing.

                        • Actions