6 Replies Latest reply on Apr 12, 2011 8:26 AM by peterj

JBoss6: java.lang.OutOfMemoryError: PermGen space

nizhnegorsliy Apr 8, 2011 2:42 AM

Hi guys!

After Jboss5.1->Jboss6 migration I have the following error (after 5-7 deploys):

10:20:20,133 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] Error installing to PostClassLoader: name=vfs:///home/kchepyrev/jboss-6.0.0.Final/server/nmng/deploy/nmng.ear state=ClassLoader mode=Manual requiredState=PostClassLoader: java.lang.OutOfMemoryError: PermGen space
          at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:185) [:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832) [:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550) [:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1603) [:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491) [:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983) [:2.2.0.GA]
          at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076) [:2.2.0.GA]
          at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679) [:2.2.0.GA]
          at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106) [:6.0.0.Final]
          at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143) [:0.2.2]
          at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.deploy(HDScanner.java:240) [:0.2.2]
          at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.complete(HDScanner.java:192) [:0.2.2]
          at org.jboss.profileservice.management.TwoPCActionWrapper.doComplete(TwoPCActionWrapper.java:57) [:0.2.2]
          at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.complete(AbstractTwoPhaseModificationAction.java:74) [:0.2.2]
          at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.prepare(AbstractTwoPhaseModificationAction.java:95) [:0.2.2]
          at org.jboss.profileservice.management.ModificationSession.prepare(ModificationSession.java:87) [:0.2.2]
          at org.jboss.profileservice.management.AbstractActionController.internalPerfom(AbstractActionController.java:234) [:0.2.2]
          at org.jboss.profileservice.management.AbstractActionController.performWrite(AbstractActionController.java:213) [:0.2.2]
          at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:150) [:0.2.2]
          at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:135) [:0.2.2]
          at org.jboss.profileservice.deployment.hotdeploy.HDScanner.scan(HDScanner.java:146) [:0.2.2]
          at org.jboss.profileservice.deployment.hotdeploy.HDScanner.run(HDScanner.java:90) [:0.2.2]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_24]
          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_24]

Then, I changed JAVA_OPTS:

JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC 50 # -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled"

But I recived another error:

Scan failed: java.lang.VerifyError: (class: org/jboss/deployers/spi/DeploymentException, method: rethrowAsDeploymentException signature: (Ljava/lang/String;Ljava/lang/Throwable;)Lorg/jboss/deployers/spi/DeploymentException;) Illegal constant pool index
at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:1370) [:2.2.0.GA]
at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:1316) [:2.2.0.GA]
at org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:968) [:2.2.0.GA]
at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.checkComplete(MainDeployerPlugin.java:82) [:6.0.0.Final]
at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.checkComplete(ProfileControllerContext.java:138) [:0.2.2]
at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.deploy(HDScanner.java:246) [:0.2.2]
at org.jboss.profileservice.deployment.hotdeploy.HDScanner$HDScanAction.complete(HDScanner.java:192) [:0.2.2]
at org.jboss.profileservice.management.TwoPCActionWrapper.doComplete(TwoPCActionWrapper.java:57) [:0.2.2]
at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.complete(AbstractTwoPhaseModificationAction.java:74) [:0.2.2]
at org.jboss.profileservice.management.actions.AbstractTwoPhaseModificationAction.prepare(AbstractTwoPhaseModificationAction.java:95) [:0.2.2]
at org.jboss.profileservice.management.ModificationSession.prepare(ModificationSession.java:87) [:0.2.2]
at org.jboss.profileservice.management.AbstractActionController.internalPerfom(AbstractActionController.java:234) [:0.2.2]
at org.jboss.profileservice.management.AbstractActionController.performWrite(AbstractActionController.java:213) [:0.2.2]
at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:150) [:0.2.2]
at org.jboss.profileservice.management.AbstractActionController.perform(AbstractActionController.java:135) [:0.2.2]
at org.jboss.profileservice.deployment.hotdeploy.HDScanner.scan(HDScanner.java:146) [:0.2.2]
at org.jboss.profileservice.deployment.hotdeploy.HDScanner.run(HDScanner.java:90) [:0.2.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [:1.6.0_24]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [:1.6.0_24]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [:1.6.0_24]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [:1.6.0_24]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [:1.6.0_24]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

Any ideas? Please help!

1. Re: JBoss6: java.lang.OutOfMemoryError: PermGen space

sashka_nem Apr 8, 2011 2:53 AM (in response to nizhnegorsliy)

I can't see param to increase pernament generation memory size, of your VM.
Return to default what you have changed and try adding this param: -XX:MaxPermSize=256M (256 or whatever)

You can use big permgen size for development purpose, to handle many hot deployments, but in production it must me more intuitive.
Actions
2. Re: JBoss6: java.lang.OutOfMemoryError: PermGen space

shelly.mcgowan Apr 8, 2011 8:29 AM (in response to sashka_nem)

If your application includes JSF, see JBAS-8443 .
Actions
3. Re: JBoss6: java.lang.OutOfMemoryError: PermGen space

nizhnegorsliy Apr 11, 2011 8:44 AM (in response to shelly.mcgowan)

I have several applications, some of them doesn't includes JSF. Also, I replaced $JBOSS_HOME/server/$my_conf/deployers/jsf.deployer/Mojarra-2.0/jsf-libs/jsf-*.jar files (and I see "Initializing Mojarra 2.1.1..." messages when starting Jboss), but OOME still present.

upd. My current JAVA_OPTS is:

JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Xms256M -Xmx512M -XX:MaxPermSize=256M "
Actions
4. Re: JBoss6: java.lang.OutOfMemoryError: PermGen space

peterj Apr 11, 2011 11:03 AM (in response to nizhnegorsliy)

Try increasing MaxPermSize to 512M. You might also monitor the amount of permgen used, perhaps by monitoring the size you can notice when there is an increase in the size and related that to activities taking place at that time.

Also, I would never use the CMS collector with less than a 1024MB heap, and I would set both -Xms and -Xmx to 1024M. And how many cores are on this machine? I wouldn't use CMS with less than 4 cores. If the CMS collector gets behind on cleaning the garbage you will be in a lot of pain.
Actions
5. JBoss6: java.lang.OutOfMemoryError: PermGen space

nizhnegorsliy Apr 12, 2011 2:36 AM (in response to peterj)

I changed JAVA_OPTS like this:
JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+Use ConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Xms1024M -Xmx1024M -XX:MaxPermSize=512m "
but the problem still present.
My machine for developement has 2 cores and the productive machine has 12 cores.
Actions
6. JBoss6: java.lang.OutOfMemoryError: PermGen space

peterj Apr 12, 2011 8:26 AM (in response to nizhnegorsliy)

1) Is the OOME showing up in development or in production?

2) Are you starting to do the monitoring that I suggested?
Actions

Go to original post