11 Replies Latest reply on Feb 15, 2012 6:09 AM by guinotphil

    ClassLoader Memory Leaks on AS7

    guinotphil

      Hello,

       

      A few weeks ago I noticed that after many deploy / undeploy of my application (a Seam 2 application using RestEasy and Hibernate 3) I ended with a OutOfMemoryError: PermGen space (cf. https://community.jboss.org/message/713928 ).

       

       

      before.png

      At each deployment the PermGen was more and more used, and no classes were unloaded.

       

      After some investigations with Eclipse Memory Analyzer I found out this was due to Class Loader Memory Leaks, mainly because of created threads, usages of ThreadLocal or WeakHashMap without SoftReferences having the value referencing the key…

       

       

      Here are the different points that I found causing leaks on my application. Each of them is a different issue, and I’ve been testing (with last week's nightly build) the deploy/undeploy without only one of each, to be sure that it was a real issue.

       

      Application ClassLoader leaks:

      HibernateAnnotationScanner keeps references to application’s ClassLoader (minor issue): https://issues.jboss.org/browse/AS7-3734

      javax.ws.rs.ext.FactoryFinder use application’s Class Loader which causes CL Memory Leak: https://issues.jboss.org/browse/AS7-3735

      javax.el.BeanELResolver.properties keeps references to undeployed classes: https://issues.jboss.org/browse/AS7-3736

       

      Threads ContextClassLoader leaks:

      AWT AppContext / EventQueue ClassLoader Memory Leaks: https://issues.jboss.org/browse/AS7-3733

      sun.net.www.http.KeepAliveCache preventing classloader from being garbage collected: https://issues.jboss.org/browse/AS7-3732

      Threads ContextClassLoader leak caused by EJB passivation pool: https://issues.jboss.org/browse/AS7-3737

       

      ThreadLocal pseudo-leaks:

      org.jboss.resteasy.util.ThreadLocalStack creates ThreadLocal pseudo-leak on AS 7: https://issues.jboss.org/browse/RESTEASY-660

       

       

      After all these patches applied, I can deploy and undeploy my application and with Eclipse Memory Analyzer I can see that I no longer have any path from GC Root to my application’s classes which are not Weak/Soft/Phantom.

       

      Sadly, with a quite low PermGen to be honest, after a couple of deployments I ended up again with a PermGen space OutOfMemoryError. Despite not having any strong references to my undeployed classes, the undeployed application was not being garbage collected. It seems that because of the SoftReferences the GC will consider removing them only when the available HeapSpace becomes low, not the PermGen space. And when the application gets deployed again, it’s not really the time when the HeapSpace becomes quite loaded.

      The only work around I’ve found to call the garbage collector removing those SoftReferences is to add the JVM option:

      -XX:SoftRefLRUPolicyMSPerMB=1

      and to load the heap space at a high usage, so the GC will consider cleaning the soft references:

      final int ONE_MB = 1024*1024;

      final int NB_MB = 100;

       

      final Runtime runtime = Runtime.getRuntime();

       

      final List<byte[]> data = new ArrayList<byte[]>();

      try {

          while (runtime.freeMemory() > NB_MB*ONE_MB) {

             data.add(new byte[ONE_MB*NB_MB / 2]);

             runtime.gc();

             runtime.runFinalization();

             try {

                 Thread.sleep(NB_MB);

             } catch (InterruptedException e) {

                 throw new RuntimeException(e);

             }

          }

      } catch (OutOfMemoryError e) {

          log.debug("Out of memory error", e);

      }

       

      try {

                   Thread.sleep(2*NB_MB);

      } catch (InterruptedException e) {

                   throw new RuntimeException(e);

      }

       

      data.clear();

      runtime.gc();

       

      This is very ugly I know (ie. I don’t recommend using it a production, a restart of the server would be the best, as a bigger PermGen would only delay the problem), but I don’t think there are other way to make the JVM cleaning soft references.

       

      Then, at deployment the classes are unloaded and the PermGen has more free space:

      after.png

        • 1. Re: ClassLoader Memory Leaks on AS7
          smarlow

           

          After all these patches applied, I can deploy and undeploy my application and with Eclipse Memory Analyzer I can see that I no longer have any path from GC Root to my application’s classes which are not Weak/Soft/Phantom.

           

          Sadly, with a quite low PermGen to be honest, after a couple of deployments I ended up again with a PermGen space OutOfMemoryError. Despite not having any strong references to my undeployed classes, the undeployed application was not being garbage collected. It seems that because of the SoftReferences the GC will consider removing them only when the available HeapSpace becomes low, not the PermGen space. And when the application gets deployed again, it’s not really the time when the HeapSpace becomes quite loaded.

           

           

          I made a different fix to the Hibernate adapters that might help reduce the number of references that need to be cleaned up.  At this point, I'm not sure if we will fix in AS 7.1 or a little later.  It might be helpful to know if it helps you or not.  The changes are referenced from the pull request at https://github.com/jbossas/jboss-as/pull/1506.

          • 2. Re: ClassLoader Memory Leaks on AS7
            smarlow

            For the ContextClassLoader leak caused by EJB passivation pool: https://issues.jboss.org/browse/AS7-3737, it might be better to clear the thread context classloader (after pushing tccl) before creating the new GroupAwareBackingCacheImpl (and then restore the tccl). 

            • 3. Re: ClassLoader Memory Leaks on AS7
              smarlow

              Do you have a unit test case or standalone test application that recreates the leaks?

              • 4. Re: ClassLoader Memory Leaks on AS7
                pferraro

                FYI - I've submitted a pull requst for AS7-3737.

                https://github.com/jbossas/jboss-as/pull/1513

                An ejb cache will now properly shutdown its scheduled executor on undeploy.

                • 5. Re: ClassLoader Memory Leaks on AS7
                  smarlow

                  Could you test again with AS7 nightly build # 605 or greater (http://community.jboss.org/thread/167590).

                   

                  Paul's fix for AS-3737 is in and the JPA change as well. 

                  • 6. Re: ClassLoader Memory Leaks on AS7
                    smarlow

                    Actually, looks like build # 605 was bad (grab a newer one than that). 

                    • 7. Re: ClassLoader Memory Leaks on AS7
                      smarlow

                      The builds from yesterday are still catching up.  Build # 609 has the JPA fix but the EJB passivation change is still in queue. 

                      • 8. Re: ClassLoader Memory Leaks on AS7
                        guinotphil

                        Thank you.

                         

                        The JPA issue was not a major issue... As I said at the end of the day the reference to the application CL won't be kept, but I noticed that the weak hash map was not expunged at undeploy. This caused less memory from the perm gen set free. I actually noticed that as it was only at the 3rd deployement that the classes get unloaded, while on the graph I posted above, the 2nd deployement caused the classes being unloaded.

                         

                         

                        The AWT - which as been closed on JIRA - can not be solved by adding any listener in the application. Originally this is because of Hibernate 3's EventListenerConfigurator which use java.beans.introspector which depends on AppContext. At the point the AppContext will be created and will keep a final reference to the application classloader. Some server-side code should be run at startup to avoid all the AWT's classes keeping references to any deployed application.

                        • 9. Re: ClassLoader Memory Leaks on AS7
                          smarlow

                          The AWT - which as been closed on JIRA - can not be solved by adding any listener in the application. Originally this is because of Hibernate 3's EventListenerConfigurator which use java.beans.introspector which depends on AppContext. At the point the AppContext will be created and will keep a final reference to the application classloader. Some server-side code should be run at startup to avoid all the AWT's classes keeping references to any deployed application.

                           

                          That sounds like one possible workaround, to eagerly initialize code that keeps a reference to its first callers classloader.  How to package that code and where it comes from, I'm not sure yet.  Is it an application that is started before other apps or a subsystem or something else.  Or is it something that could  be fixed without such a hack (via changes in existing code).

                           

                          Could you attach a picture of the Eclipse MAT with the AWT leak?  Would be nice to see pictures of the other leaks also (except the ejb passivation/jpa ones which should be fixed).

                           

                          I assume your heapdump is full of your private data, otherwise I would ask for a copy of that.

                          • 10. Re: ClassLoader Memory Leaks on AS7
                            guinotphil

                            Hello,

                             

                            Here is the the details about the AppContext leak. As you can see it keeps a reference here to a JBoss module CL which is actually the Cl of the application.

                             

                            01.png

                             

                            However, this leak does not happen everytime at deploy.

                             

                            Sometimes AppContext is first called

                            at java.beans.Introspector.flushFromCaches(Introspector.java:349) [rt.jar:1.6.0_30]

                            at org.hibernate.ejb.EventListenerConfigurator.configure(EventListenerConfigurator.java:262)

                            at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:1100)

                            at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:689)

                            at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:73)

                            at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:162)

                            at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.start(PersistenceUnitServiceImpl.java:85)

                            at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                            at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]

                            at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]

                             

                            in that case, it will cause the leak.

                             

                            But sometimes the first call to AppContext is

                            at javax.imageio.spi.IIORegistry.getDefaultInstance(IIORegistry.java:137) [rt.jar:1.6.0_30]

                            at javax.imageio.ImageIO.<clinit>(ImageIO.java:48) [rt.jar:1.6.0_30]

                            at org.ajax4jsf.resource.ResourceBuilderImpl.<clinit>(ResourceBuilderImpl.java:106) [richfaces-impl-3.3.3.Final.jar:3.3.3.Final]

                            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.6.0_30]

                            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) [rt.jar:1.6.0_30]

                            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) [rt.jar:1.6.0_30]

                            at java.lang.reflect.Constructor.newInstance(Constructor.java:513) [rt.jar:1.6.0_30]

                            at java.lang.Class.newInstance0(Class.java:355) [rt.jar:1.6.0_30]

                            at java.lang.Class.newInstance(Class.java:308) [rt.jar:1.6.0_30]

                            at org.ajax4jsf.resource.InternetResourceBuilder.getInstance(InternetResourceBuilder.java:167) [richfaces-api-3.3.3.Final.jar:3.3.3.Final]

                            at org.ajax4jsf.renderkit.ChameleonRenderKitFactory.<init>(ChameleonRenderKitFactory.java:62) [richfaces-impl-3.3.3.Final.jar:3.3.3.Final]

                            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.6.0_30]

                            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) [rt.jar:1.6.0_30]

                            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) [rt.jar:1.6.0_30]

                            at java.lang.reflect.Constructor.newInstance(Constructor.java:513) [rt.jar:1.6.0_30]

                            at javax.faces.FactoryFinder.getImplGivenPreviousImpl(FactoryFinder.java:519) [jsf-api-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at javax.faces.FactoryFinder.getImplementationInstance(FactoryFinder.java:405) [jsf-api-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at javax.faces.FactoryFinder.access$400(FactoryFinder.java:135) [jsf-api-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at javax.faces.FactoryFinder$FactoryManager.getFactory(FactoryFinder.java:717) [jsf-api-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at javax.faces.FactoryFinder.getFactory(FactoryFinder.java:239) [jsf-api-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at com.sun.faces.config.processor.FactoryConfigProcessor.verifyFactoriesExist(FactoryConfigProcessor.java:187) [jsf-impl-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at com.sun.faces.config.processor.FactoryConfigProcessor.process(FactoryConfigProcessor.java:132) [jsf-impl-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:205) [jsf-impl-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:200) [jsf-impl-1.2_15-jbossorg-2.jar:1.2_15.jbossorg-1-20111019-SNAPSHOT]

                            at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:3392) [jbossweb-7.0.10.Final.jar:]

                            at org.apache.catalina.core.StandardContext.start(StandardContext.java:3850) [jbossweb-7.0.10.Final.jar:]

                            at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90) [jboss-as-web-7.1.0.Final-SNAPSHOT.jar:7.1.0.Final-SNAPSHOT]

                            at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)

                            at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]

                            at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]

                             

                            And it that case it seems that the AppContext will refer only to the System class loader..

                             

                             

                            I'm not really sure why the first call is different, I think this is because of upgrading to last week nightly build to the last one yesterday...

                            • 11. Re: ClassLoader Memory Leaks on AS7
                              guinotphil

                              And here are the detail of the leak caused by the EventQueue.

                              As you can see the EventQueue keep a reference to the class loader, and also a Thread is created.

                               

                              02.png

                              (click on the image to see it at proper size)

                               

                              Calling AppContext is not enough. But a call to Toolkit.getDefaultToolkit() seems to do the job for both leaks.