ModuleClassLoader does not release application classes on undeploy.
stvhl Apr 10, 2013 9:58 AMHi,
If I run an instance of the JBoss (version 7.1.1.Final) application server and deploy my EAR-bundled application to it, then undeploy it, JBoss does not release the classes of the application and they're never garbage collected. Constant deploying and redeploying of the application will eventually lead to a PermGen space exception.
A few facts about my application:
- I'm using Maven to build a WAR, which is then wrapped in an EAR. The EAR is then deployed to the application server.
- I'm using libraries that are provided by the application server, specifically Hibernate, Hibernate Envers and some of the javax classes. I have marked these dependencies as 'provided' in Maven, so that they aren't included in the bundled EAR. I let JBoss provide these libraries for me. Any optional libraries (such as Hibernate Envers) are brought in by adding the depedencies tag in the MANIFEST file of the WAR.
I've investigated this issue and arrived at the following steps:
- For each iteration, I will start a standalone instance, deploy and undeploy. I then take a Heap snapshot and analyze it in the Memory Analyzer for Eclipse.
- The first iteration highlighted that the ModuleClassLoader was being referenced by a dom4j library. I fixed this issue by adding a Maven dependency to dom4j and marking it as provided, meaning that JBoss will supply the JAR. I believe this bug is documented here: http://sourceforge.net/p/dom4j/bugs/108/
- The second iteration highlighted an issue with Hibernate Envers. I believe there are work arounds to this error, but for the time being I have removed a reference to Hibernate Envers (I believe it's configured wrong).
- Once the dom4j and Envers issues had been removed, I still find the leak. The difference is that now the ModuleClassLoader is referenced by an incredibly large chain of java.lang.ref.Finalizer objects, contained within a Timer in a TimerThread.
The leak is consistent in that if I deploy the application and undeploy it 3 times, I find that there are 3 ModuleClassLoaders all taking up roughly the same amount of Heap space.
I'm completely lost as to what might be keeping a reference, as I've trawled through our code to highlight any static references and other bad smells to memory leak, but nothing seems to change this huge Finalizer chain.
A few things I've noticed about the Finalizer chain:
- If I trace the Finalizer chain back the root, I find a Finalizer with an unfinalized field which references the first Finalizer. The first finalizer has a reference to sun.net.www.http.KeepAliveStream, which references a Finalizer with referent java.io.FileOutputStream
- from then on, each Finalizer references another Finalizer with the "referent" of either:
- java.util.zip.ZipFile$ZipFileInputStream
- java.util.zip.ZipFile$ZipFileInflaterInputStream
- eventually one Finalizer's referent is org.jboss.logmanager.LoggerNode which references a Timer, which references a TimerThread, which references the ModuleClassLoader through its "contextClassLoader" field
It honestly seems like something is trying to finalize and is being blocked, and as there are a few references to ZipFile streams, maybe it's a stream that isn't properly closed? We aren't using streams or anything to do with zip files in our application.
Apologies if the post seems a bit all over the place, but I guess that correctly represents my brain right now. I'd be more than happy to provide you with as much information you need from the logs I have, just ask!
I've attached a ZIP file containing Memory Analyzer's Leak Suspects report.
Thanks for any help you can provide.