Classloader leak in BeanMetaDataCache?

[hwellmann.de]

I'm currently experimenting with JBoss AS 7, using a real-world Java EE 6 application which I've already tested on JBoss AS 6, GlassFish and Resin. The application uses JPA, EJB, CDI, JAXB, Servlet and JSP.

After redeploying the application a couple of times by simply touching the WAR in the standalone/deployments directory, I'm getting a PermGen OutOfMemoryError, which did not occur on the other servers.

Inspecting a heap dump with Eclipse Memory Analyzer, I can see a single instance of org.hibernate.validator.metadata.BeanMetaDataCache holding references to various classes from my own persistence unit, but these classes are loaded from different instances of org.jboss.modules.ModuleClassLoader, all with the same module identifier deployment.myapp.war.

Could it be that the BeanMetaDataCache does not get cleared when an application is undeployed, so that the server core keeps holding stale references to application classloaders?

Best regards,

Harald

[smarlow]

Could you attach some screen shots of what you see in Eclipse Memory Analyzer (maybe from the "find GC root" with all weak/soft/not reachable refences ignored)?  The heapdump contains a copy of your application memory state, so probably best to not share the heap dump.

Sure sounds like you have found a bug (org.hibernate.validator.metadata.BeanMetaDataCache) that could cause the PergGen OOM.

[smarlow]

I see a potential cause  in the AS7 JPA subsystem.  Could you create a jira (https://issues.jboss.org/browse/AS7) issue for this.

Thanks for reporting the bug!

Scott

[smarlow]

Harald,

I would like to give you a chance to create the jira still for this bug.  If not, I'll create it.

Thanks again for finding this and reporting!

Scott

[hwellmann.de]

Scott, thanks for looking into this. Here's the JIRA issue: https://issues.jboss.org/browse/AS7-2251.

Best regards,

Harald

[smarlow]

Potential fix pushed to https://github.com/scottmarlow/jboss-as/commits/validatorleak

[guinotphil]

Hello,

I'm running with a similar issue:  PermGen OutOfMemoryError after many deployments.

Actually, the server does not remove the webapp loaded classes from the PermGen and the number of loaded classes and the PermGen memory increases.

I’ve used to Eclipse Memory Analyzer to find out why the classes where still referenced, and I’ve found out that the web application class loader (which contains references to all the application’s classes which contains references to all the static fields) is set on JVM’s sun.awt.AppContext class’ static field mainAppContext.

After some investigations (simply removing the class from the JVM to find out when it get called), I’ve found out this mainAppContext is set during the initialization which is called during hibernate’s configuration:

org.jboss.msc.service.StartException in service jboss.persistenceunit."MyEAR.ear#entityManagerFactory": Failed to start service

          at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1780) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_30]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_30]

          at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]

Caused by: java.lang.NoClassDefFoundError: sun/awt/AppContext

          at java.beans.Introspector.flushFromCaches(Introspector.java:349) [:1.6.0_30]

          at org.hibernate.ejb.EventListenerConfigurator.configure(EventListenerConfigurator.java:262)

          at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:1100)

          at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:689)

          at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:73)

          at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:157)

          at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.start(PersistenceUnitServiceImpl.java:81)

          at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1824) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

          at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1759) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

I guess a work around would be that at startup AS load the AppContext so it’d never refer to a web application class loader.

[smarlow]

Steve created a jira  for this issue.  You might try the code change that he mentioned and submit a patch for it.

[smarlow]

You also could try using Hibernate 4.0.1.

[guinotphil]

No, I use the latest version of Hibernate 3, and I won't be able to use Hibernate 4.

I'm afraid, I think this is a different issue, and more related to AS 7 than Hibernate.

I also noticed that the EJB pool's threads hold a reference to the web-app's classloaders even after the undeployement. This cause PermGen OutOfMemoryError.

[smarlow]

Can you recreate the leak with the nightly AS7 build? The link is here.  That will tell us if your hitting an already solved problem or not.

[guinotphil]

I'm using nightly built from 27th January, but I'll try with the latest one and give you all the details as soon as I'll have identified all the issues...