4 Replies Latest reply on Apr 10, 2013 11:37 AM by stvhl

    ModuleClassLoader does not release application classes on undeploy.

    stvhl

      Hi,

       

      If I run an instance of the JBoss (version 7.1.1.Final) application server and deploy my EAR-bundled application to it, then undeploy it, JBoss does not release the classes of the application and they're never garbage collected. Constant deploying and redeploying of the application will eventually lead to a PermGen space exception.

       

      A few facts about my application:

      • I'm using Maven to build a WAR, which is then wrapped in an EAR. The EAR is then deployed to the application server.
      • I'm using libraries that are provided by the application server, specifically Hibernate, Hibernate Envers and some of the javax classes. I have marked these dependencies as 'provided' in Maven, so that they aren't included in the bundled EAR. I let JBoss provide these libraries for me. Any optional libraries (such as Hibernate Envers) are brought in by adding the depedencies tag in the MANIFEST file of the WAR.

       

      I've investigated this issue and arrived at the following steps:

      • For each iteration, I will start a standalone instance, deploy and undeploy. I then take a Heap snapshot and analyze it in the Memory Analyzer for Eclipse.
      • The first iteration highlighted that the ModuleClassLoader was being referenced by a dom4j library. I fixed this issue by adding a Maven dependency to dom4j and marking it as provided, meaning that JBoss will supply the JAR. I believe this bug is documented here: http://sourceforge.net/p/dom4j/bugs/108/
      • The second iteration highlighted an issue with Hibernate Envers. I believe there are work arounds to this error, but for the time being I have removed a reference to Hibernate Envers (I believe it's configured wrong).
      • Once the dom4j and Envers issues had been removed, I still find the leak. The difference is that now the ModuleClassLoader is referenced by an incredibly large chain of java.lang.ref.Finalizer objects, contained within a Timer in a TimerThread.

       

      The leak is consistent in that if I deploy the application and undeploy it 3 times, I find that there are 3 ModuleClassLoaders all taking up roughly the same amount of Heap space.

       

      I'm completely lost as to what might be keeping a reference, as I've trawled through our code to highlight any static references and other bad smells to memory leak, but nothing seems to change this huge Finalizer chain.

       

      A few things I've noticed about the Finalizer chain:

      • If I trace the Finalizer chain back the root, I find a Finalizer with an unfinalized field which references the first Finalizer. The first finalizer has a reference to sun.net.www.http.KeepAliveStream, which references a Finalizer with referent java.io.FileOutputStream
      • from then on, each Finalizer references another Finalizer with the "referent" of either:
        • java.util.zip.ZipFile$ZipFileInputStream
        • java.util.zip.ZipFile$ZipFileInflaterInputStream
      • eventually one Finalizer's referent is org.jboss.logmanager.LoggerNode which references a Timer, which references a TimerThread, which references the ModuleClassLoader through its "contextClassLoader" field

       

      It honestly seems like something is trying to finalize and is being blocked, and as there are a few references to ZipFile streams, maybe it's a stream that isn't properly closed? We aren't using streams or anything to do with zip files in our application.

       

      Apologies if the post seems a bit all over the place, but I guess that correctly represents my brain right now. I'd be more than happy to provide you with as much information you need from the logs I have, just ask!

       

      I've attached a ZIP file containing Memory Analyzer's Leak Suspects report.

       

      Thanks for any help you can provide.

        • 1. Re: ModuleClassLoader does not release application classes on undeploy.
          stvhl

          Some more information.

           

          After drilling down the full Finalizer queue, I noticed that the first Finalizer's referent is a java.io.FileOutputStream. Next in the queue is a FileInputStream, then another OutputStream, then a ZipFileInflater and so on..

           

          Could it be that the FileOutputStream's finalize method is being blocked, which is causing the remainder of objects not to proceed?

          • 2. Re: ModuleClassLoader does not release application classes on undeploy.
            wdfink

            Could you test it with EAP6.1.0.Alpha as 7.1.1 will be old and the bug might be fixed.

            • 3. Re: ModuleClassLoader does not release application classes on undeploy.
              stvhl

              Yeah, I can do that. Unfortunately the choice of JBoss version number is our deployment environment and it's not my decision whether we use it. Similarly, we have other applications that deploy to the same server using roughly the same libraries and they don't suffer the same problem, so clear it is out configuration.

               

              Upon further investigation

              I waited quite a while (maybe 30mins to an hour) and took another snapshot of the JBoss java process. I noticed that the Heap size had reduced. The ModuleClassLoader was still being referenced (and as an effect, so were all the application classes), but the TimerThread had disappeared. I drilled down the Finalizer queue once again and at the bottom found a Finalizer for the class org.jboss.logmanager.LoggerNode.

               

              Could it be that the LoggerNode is not finalizing properly? I took a look at the LoggerNode source and found that it tries to obtain a lock for the log, as it needs to set its own level to null. This could be holding up the application. Does it sound familiar, or can you point me in the right direction? It may be a case that my logging is not configured properly and I have a concurrency issue when LoggerNode tries to finalize, at least that's my best guess at the moment.

               

              As for the FileOutputStream mentioned in the original post, I now suspect that was either writing to a file or waiting for a file to close, and waiting some time before taking another Heap snapshot took care of that.

               

              I'll get back to you on the EAP6.1.0.Alpha test, though I'd still appreciate some input regarding the LoggerNode.

              • 4. Re: ModuleClassLoader does not release application classes on undeploy.
                stvhl

                I deployed it on the newer version as you said. When I undeployed and took a heap snapshot (<10 seconds after undeploy) I still saw my classes.

                 

                This time the ModuleClassLoader was held by a reference in the following chain (all in the org.jboss.logmanager package):

                 

                LogContext:logContextSelector > ThreadLocalLogContextSelector:delegate > ClassLoaderLogContextSelector:contextMap > CopyOnWriteMap:map > FastCopyHashMap:table > entry

                 

                Again, this is pointing to the logmanager. Does anyone have experience with this? Presumably I'm configuring my logging in an incorrect way such that it won't release its reference to the classes I'm trying to log. That in turn references the class loader. Any ideas?

                 

                I'll do my usual wait an hour and take another heap snapshot to see if it's an issue with GC cycles, but I don't see any huge Finalizer queues this time so I don't think the application is hanging on one Finalize.

                 

                Still interested in anwers to my previous post regarding the LoggerNode's finalize method.