Memory leak and PermGen error

[oneworld95]

Hi. We're getting this error in our logs when we redeploy a Seam app:

2009-07-02 10:11:30,745 WARN  [org.jboss.system.ServiceController] Problem starting service jboss.we
b.deployment:war=forms_new_employee.war,id=-1855897005
java.lang.OutOfMemoryError: PermGen space
....
2009-07-02 10:11:59,745 ERROR [STDERR] java.lang.OutOfMemoryError: PermGen space

This error happens after some 10 or 12 redeployments. We've bumped up the memory assigned to the apps but this problem occurs again. Some have suggested that the PermGen error is caused by memory leaks in the Seam app. How do we track down the memory leak? Any suggestions/thoughts on this error are appreciated.

Thank you for your help in advance.

[luxspes]

This is an intrinsic JVM problem. The only way to workaround it is to increase the PermGen size or to use a JVM without the PermGen limitation (JRockit, but it is only free for development, not for production purposes, I wonder if that will change now that Sun and Oracle are the same company).

Mmmm, I wonder if OSGi module loading/unloading is affected by this...

[luxspes]

Try with this:

-server -XX:MaxPermSize=128m -XX:+UseParNewGC -XX:MaxNewSize=256m -XX:NewSize=256m -Xms768m -Xmx768m -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0  -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

but after a lot of googling I am not sure it actually works (some people report it does, some that it does not)

[oneworld95]

Thank you, Francisco. Increasing the memory only delays the inevitable. Various posts talk about how this as a ClassLoader issue: Sun Blog

Even some Apache Commons classes can cause this; it seems to occur with Static objects, used across the application, and which the Garbage Collector is unable to clean up.

Here are some pertinent info from the article:

...any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak.

If there's a problem with garbage collecting classes and if you keep loading new classes, the VM will run out of space in that memory region, even if there's plenty of memory available on the heap. Setting the -Xmx parameter will not help: this parameter only specifies the size of the total heap and does not affect the size of the PermGen region.

JDK 6 has tools to try to track down the offending class, but it's not easy. And the Sun JVM has no way of turning off the PermGen. There is a way to track the memory usage and perhaps do something before JBoss crashes: Detect JVM Memory Usage

For many of us, we'll have to live with this and simply restart JBoss when it crashes. The JRockIt JVM can't be used on production for free so it's not an option for us. I wish there was a way to turn off the PermGen feature of the Sun JVM, but it's intrinsic to how it works.

Sigh.

[luxspes]

What we need is an MVM support for Java! Sadly the JSR 121: Application Isolation API Specification looks abandoned since 2005.

MVM would allow to really run application running inside a JEE server without risk of affecting the memory of the application server (it is sad that both IIS and JavaScript support Process Isolation, but JEE app servers do not).

Some of the expensive JavaEnterpriseEdition ApplicationServers solve this problem by starting an independant Java process for each application, with OC4J web based managment ui it is just a matter of a few clicks and you can have each of your applications running in a separate JVM process, but that also has its price

[norman]

Just FYI, all of the memory issues went away for me with JBoss 5.

[sherkan777]

Can u post Norman what is so cool in jboss 5, that all memory issues went away?

[oneworld95]

I'm curious as well as to how your memory issues disappeared with JBoss 5.

[norman]

I don't have any technical explanation as I have not investigated.  I only know that with JBoss 4 I could not survive very many redeploys before getting memory issues.  I have yet to see a permgen error since switching to JBoss 5.  I doubt it's a coincidence.

[asookazian]

Norman Richards wrote on Jul 07, 2009 19:03:


I don't have any technical explanation as I have not investigated.  I only know that with JBoss 4 I could not survive very many redeploys before getting memory issues.  I have yet to see a permgen error since switching to JBoss 5.  I doubt it's a coincidence.

Norman, post your JVM and version (i.e. Sun HotSpot JDK 1.6, etc.) as well as your JVM tuning.  This is critical.

Dan Allen, in SiA, cites a JVM tuning that apparently he never has problems with:

To avoid an untimely death of the JVM, and in turn, JBoss AS, you should supply the
following parameters in the VM options of the JBoss AS runtime configuration in your
IDE (or bin/run.conf in the JBoss AS home directory). I have never experienced an
out-of-memory error when using these settings.

-Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m
-Xverify:none

I have used the following with EAP 4.3_CP02 with little or no PermGen problems in Eclipse:

-server -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000
 -Dsun.rmi.dgc.server.gcInterval=3600000
 -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
 -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512m
 -Xverify:none

Release ID: JBoss [EAP] 4.3.0.GA_CP02 (build: SVNTag=JBPAPP_4_3_0_GA_CP02 date=200808051050)

[norman]

I don't do anything special to the JVM for dev.  When using JBoss 4.2, I had to do all those crazy JVM args to help keep the server up.  But, I now I just sh run.sh.  JDK version, since you asked:

$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)

I'm not saying that nobody will encounter memory issues with JBoss 5.   But my experience, having been on JBoss 5 for quite some time now doing normal Seam development is that all my memory issues are gone.

[asookazian]

Since the PermGen out of space error has to do with class loading (and usually re-deploying your app many times per session), this indicates that the JBoss and/or other class loaders that are associated with the AS 5 libraries have been refactored/optimized to help alleviate this problem.

Is there a JIRA on this class loading optimization in AS 5.x that we can verify?

[oneworld95]

Thank you for all the ideas. I need to try the JVM parameters and see if they fix the issue.

[oneworld95]

Here are the settings in our run.conf file, but they don't prevent the issue:

-Xms1g -Xmx1g -XX:PermSize=64m -XX:MaxPermSize=256m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

Any other ideas?

[asookazian]

What JDK, Seam, JBoss versions are you using and what exactly are you doing after restarting JBoss and deploying your apps?

Have you tried using jconsole?  It's in your JDK's bin directory and will show a graph of the PermGen and other spaces (i.e. after a re-deployment, you'll see the PermGen space allocation go up).