ClassLoader Memory Leaks on AS7
Hello,
A few weeks ago I noticed that after many deploy / undeploy of my application (a Seam 2 application using RestEasy and Hibernate 3) I ended with a OutOfMemoryError: PermGen space (cf. ).
At each deployment the PermGen was more and more used, and no classes were unloaded.
After some investigations with Eclipse Memory Analyzer I found out this was due to Class Loader Memory Leaks, mainly because of created threads, usages of ThreadLocal or WeakHashMap without SoftReferences having the value referencing the key…
Here are the different points that I found causing leaks on my application. Each of them is a different issue, and I’ve been testing (with last week's nightly build) the deploy/undeploy without only one of each, to be sure that it was a real issue.
Application ClassLoader leaks:
HibernateAnnotationScanner keeps references to application’s ClassLoader (minor issue): https://issues.jboss.org/browse/AS7-3734
javax.ws.rs.ext.FactoryFinder use application’s Class Loader which causes CL Memory Leak: https://issues.jboss.org/browse/AS7-3735
javax.el.BeanELResolver.properties keeps references to undeployed classes: https://issues.jboss.org/browse/AS7-3736
Threads ContextClassLoader leaks:
AWT AppContext / EventQueue ClassLoader Memory Leaks: https://issues.jboss.org/browse/AS7-3733
sun.net.www.http.KeepAliveCache preventing classloader from being garbage collected: https://issues.jboss.org/browse/AS7-3732
Threads ContextClassLoader leak caused by EJB passivation pool: https://issues.jboss.org/browse/AS7-3737
ThreadLocal pseudo-leaks:
org.jboss.resteasy.util.ThreadLocalStack creates ThreadLocal pseudo-leak on AS 7: https://issues.jboss.org/browse/RESTEASY-660
After all these patches applied, I can deploy and undeploy my application and with Eclipse Memory Analyzer I can see that I no longer have any path from GC Root to my application’s classes which are not Weak/Soft/Phantom.
Sadly, with a quite low PermGen to be honest, after a couple of deployments I ended up again with a PermGen space OutOfMemoryError. Despite not having any strong references to my undeployed classes, the undeployed application was not being garbage collected. It seems that because of the SoftReferences the GC will consider removing them only when the available HeapSpace becomes low, not the PermGen space. And when the application gets deployed again, it’s not really the time when the HeapSpace becomes quite loaded.
The only work around I’ve found to call the garbage collector removing those SoftReferences is to add the JVM option:
-XX:SoftRefLRUPolicyMSPerMB=1
and to load the heap space at a high usage, so the GC will consider cleaning the soft references:
final int ONE_MB = 1024*1024;
final int NB_MB = 100;
final Runtime runtime = Runtime.getRuntime();
final List<byte[]> data = new ArrayList<byte[]>();
try {
while (runtime.freeMemory() > NB_MB*ONE_MB) {
data.add(new byte[ONE_MB*NB_MB / 2]);
runtime.gc();
runtime.runFinalization();
try {
Thread.sleep(NB_MB);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
} catch (OutOfMemoryError e) {
log.debug("Out of memory error", e);
}
try {
Thread.sleep(2*NB_MB);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
data.clear();
runtime.gc();
This is very ugly I know (ie. I don’t recommend using it a production, a restart of the server would be the best, as a bigger PermGen would only delay the problem), but I don’t think there are other way to make the JVM cleaning soft references.
Then, at deployment the classes are unloaded and the PermGen has more free space:
