1 2 Previous Next 16 Replies Latest reply on Jul 14, 2009 8:29 PM by Alex Cougarman

    Memory leak and PermGen error

    Alex Cougarman Apprentice

      Hi. We're getting this error in our logs when we redeploy a Seam app:



      2009-07-02 10:11:30,745 WARN  [org.jboss.system.ServiceController] Problem starting service jboss.we
      b.deployment:war=forms_new_employee.war,id=-1855897005
      java.lang.OutOfMemoryError: PermGen space
      ....
      2009-07-02 10:11:59,745 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space



      This error happens after some 10 or 12 redeployments. We've bumped up the memory assigned to the apps but this problem occurs again. Some have suggested that the PermGen error is caused by memory leaks in the Seam app. How do we track down the memory leak? Any suggestions/thoughts on this error are appreciated.


      Thank you for your help in advance.

        • 1. Re: Memory leak and PermGen error
          Francisco Jose Peredo Noguez Master

          This is an intrinsic JVM problem. The only way to workaround it is to increase the PermGen size or to use a JVM without the PermGen limitation (JRockit, but it is only free for development, not for production purposes, I wonder if that will change now that Sun and Oracle are the same company).


          Mmmm, I wonder if OSGi module loading/unloading is affected by this...

          • 2. Re: Memory leak and PermGen error
            Francisco Jose Peredo Noguez Master

            Try with this:


            -server -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:MaxNewSize=256m -XX:NewSize=256m -Xms768m -Xmx768m -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0  -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
            



            but after a lot of googling I am not sure it actually works (some people report it does, some that it does not)

            • 3. Re: Memory leak and PermGen error
              Alex Cougarman Apprentice

              Thank you, Francisco. Increasing the memory only delays the inevitable. Various posts talk about how this as a ClassLoader issue: Sun Blog


              Even some Apache Commons classes can cause this; it seems to occur with Static objects, used across the application, and which the Garbage Collector is unable to clean up.


              Here are some pertinent info from the article:



              ...any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak.


              If there's a problem with garbage collecting classes and if you keep loading new classes, the VM will run out of space in that memory region, even if there's plenty of memory available on the heap. Setting the -Xmx parameter will not help: this parameter only specifies the size of the total heap and does not affect the size of the PermGen region.

              JDK 6 has tools to try to track down the offending class, but it's not easy. And the Sun JVM has no way of turning off the PermGen. There is a way to track the memory usage and perhaps do something before JBoss crashes: Detect JVM Memory Usage


              For many of us, we'll have to live with this and simply restart JBoss when it crashes. The JRockIt JVM can't be used on production for free so it's not an option for us. I wish there was a way to turn off the PermGen feature of the Sun JVM, but it's intrinsic to how it works.


              Sigh.

              • 4. Re: Memory leak and PermGen error
                Francisco Jose Peredo Noguez Master

                What we need is an MVM support for Java! Sadly the JSR 121: Application Isolation API Specification looks abandoned since 2005.


                MVM would allow to really run application running inside a JEE server without risk of affecting the memory of the application server (it is sad that both IIS and JavaScript support Process Isolation, but JEE app servers do not).


                Some of the expensive JavaEnterpriseEdition ApplicationServers solve this problem by starting an independant Java process for each application, with OC4J web based managment ui it is just a matter of a few clicks and you can have each of your applications running in a separate JVM process, but that also has its price

                • 5. Re: Memory leak and PermGen error
                  Norman Richards Expert

                  Just FYI, all of the memory issues went away for me with JBoss 5.

                  • 6. Re: Memory leak and PermGen error
                    Gregory Nikle Apprentice

                    Can u post Norman what is so cool in jboss 5, that all memory issues went away?

                    • 7. Re: Memory leak and PermGen error
                      Alex Cougarman Apprentice

                      I'm curious as well as to how your memory issues disappeared with JBoss 5.

                      • 8. Re: Memory leak and PermGen error
                        Norman Richards Expert

                        I don't have any technical explanation as I have not investigated.  I only know that with JBoss 4 I could not survive very many redeploys before getting memory issues.  I have yet to see a permgen error since switching to JBoss 5.  I doubt it's a coincidence.

                        • 9. Re: Memory leak and PermGen error
                          Arbi Sookazian Master

                          Norman Richards wrote on Jul 07, 2009 19:03:


                          I don't have any technical explanation as I have not investigated.  I only know that with JBoss 4 I could not survive very many redeploys before getting memory issues.  I have yet to see a permgen error since switching to JBoss 5.  I doubt it's a coincidence.


                          Norman, post your JVM and version (i.e. Sun HotSpot JDK 1.6, etc.) as well as your JVM tuning.  This is critical.


                          Dan Allen, in SiA, cites a JVM tuning that apparently he never has problems with:



                          To avoid an untimely death of the JVM, and in turn, JBoss AS, you should supply the
                          following parameters in the VM options of the JBoss AS runtime configuration in your
                          IDE (or bin/run.conf in the JBoss AS home directory). I have never experienced an
                          out-of-memory error when using these settings.

                          -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000
                          -Dsun.rmi.dgc.server.gcInterval=3600000
                          -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
                          -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m
                          -Xverify:none



                          I have used the following with EAP 4.3_CP02 with little or no PermGen problems in Eclipse:


                          -server -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000
                           -Dsun.rmi.dgc.server.gcInterval=3600000
                           -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
                           -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m
                           -Xverify:none



                          Release ID: JBoss [EAP] 4.3.0.GA_CP02 (build: SVNTag=JBPAPP_4_3_0_GA_CP02 date=200808051050)

                          • 10. Re: Memory leak and PermGen error
                            Norman Richards Expert

                            I don't do anything special to the JVM for dev.  When using JBoss 4.2, I had to do all those crazy JVM args to help keep the server up.  But, I now I just sh run.sh.  JDK version, since you asked:


                            $ java -version
                            java version "1.5.0_16"
                            Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
                            Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
                            



                            I'm not saying that nobody will encounter memory issues with JBoss 5.   But my experience, having been on JBoss 5 for quite some time now doing normal Seam development is that all my memory issues are gone.

                            • 11. Re: Memory leak and PermGen error
                              Arbi Sookazian Master

                              Since the PermGen out of space error has to do with class loading (and usually re-deploying your app many times per session), this indicates that the JBoss and/or other class loaders that are associated with the AS 5 libraries have been refactored/optimized to help alleviate this problem.


                              Is there a JIRA on this class loading optimization in AS 5.x that we can verify?

                              • 12. Re: Memory leak and PermGen error
                                Alex Cougarman Apprentice

                                Thank you for all the ideas. I need to try the JVM parameters and see if they fix the issue.

                                • 13. Re: Memory leak and PermGen error
                                  Alex Cougarman Apprentice

                                  Here are the settings in our run.conf file, but they don't prevent the issue:




                                  -Xms1g -Xmx1g -XX:PermSize=64m -XX:MaxPermSize=256m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false




                                  Any other ideas?

                                  • 14. Re: Memory leak and PermGen error
                                    Arbi Sookazian Master

                                    What JDK, Seam, JBoss versions are you using and what exactly are you doing after restarting JBoss and deploying your apps?


                                    Have you tried using jconsole?  It's in your JDK's bin directory and will show a graph of the PermGen and other spaces (i.e. after a re-deployment, you'll see the PermGen space allocation go up).

                                    1 2 Previous Next