OutOfMemoryError in JBoss 5.01

[yongz]

Hi,

Env:
* jdk1.5.0_19
* jboss-5.0.1.GA

We just upgraded our production server to jboss-5.0.1.GA from jboss-4.2.2.GA today, and it caused huge problem on production:

With JVM settings:

1. -Xms1300m -Xmx1300m -XX:MaxPermSize=256m

jboss-5.0.1.GA gives OOM during startup:
Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 32776 bytes for Chunk::new. Out of swap space?

While this settings works perfectly in jboss-4.2.2.GA.

2. -Xms1200m -Xmx1200m -XX:MaxPermSize=256m

jboss-5.0.1.GA starts OK, but server got OOM again when the load increased and users started to login and use the server.

3. -Xms1024m -Xm1024m -XX:MaxPermSize=256m

jboss-5.0.1.GA starts OK, but server still got OOM again when the load increased and users started to login and use the server.

3. -Xms768m -Xm768m -XX:MaxPermSize=256m

For now we mitigate the issue with the above memory settings, but we cannot stay with this heap size for long.

Does anyone have this issue before in jboss-5.0.1.GA?

We believe boss-5.0.1.GA has undergone rigorous testing, including load testing, and should not have an issue like this.

However, I have an impression that jboss-5.0.1.GA uses too much non-heap memory which prevent it from creating more threads under load. Something to do with the loading of libs in jboss-5.0.1.GA?

Thanks in advance.

[alesj]

There is this memory leak:
- https://jira.jboss.org/jira/browse/JBCL-119
but if you're not re-deploying like mad +
have explicit CL dependencies on "static" deployment,
you're not hitting this.

[jaikiran]

In addition to what Ales said - it's better to move to AS 5.1.0 instead of 5.0.1 because 5.1.0 has a lot of bug fixes.

[yongz]

Thanks a lot for the response.

but if you're not re-deploying like mad + have explicit CL dependencies on "static" deployment

Our EAR is about 37 MB, and we are using JBossws-metro.

Would you mind explaining a bit more about the re-deploying and explicit CL dependencies on "static" deployment?

Is there a way that we can avoid this?

[alesj]

"yongz" wrote:

Would you mind explaining a bit more about the re-deploying and explicit CL dependencies on "static" deployment?

Like I said, first, I doubt you're hitting this,
as explicit CL dependencies are not yet common in recent deployments.
But I might be wrong. :-)

The scenario goes like this.
You have deployments A and B, where B expresses explicit CL dependency on A
(via jboss-classloading.xml or some other CLMetaData info).
Then you re-deploy B a lot of times, meaning A's dependsOnMe info grows big ==> OOME.

[yongz]

Thanks Ales/JAIKIRAN, much appreciated.

via jboss-classloading.xml or some other CLMetaData info

The issue you described is close to what I saw, but we did not make any of the above CL changes to JBoss or in our application.

A few more questions:
1. Could you also confirm this CL issue would cause the server cannot handle load(more threads)?
2. Is there something(jars, etc) we can upgrade in AS 5.01?
3. As JAIKIRAN suggested, will this issue be fixed in 5.1.0?

Sorry about so many questions, but I have to sort this out on prod quickly.

[alesj]

"yongz" wrote:

1. Could you also confirm this CL issue would cause the server cannot handle load(more threads)?

This is a mem-leak, so it can cause a bunch of weird issues. :-(

"yongz" wrote:

2. Is there something(jars, etc) we can upgrade in AS 5.01?

You could, but it's risky as you don't exactly know how things/libs are related.
So, the best thing to do is to use the latest version of AS.

"yongz" wrote:

3. As JAIKIRAN suggested, will this issue be fixed in 5.1.0?

5.1.0 is already out, whereas this was just recently found and fixed.
It will be part of upcoming AS release, 5.2.x.

[yongz]

Hi Ales,

Unfortunately I was able to reproduce this issue on 5.1.0 with:

-Xms1300m -Xmx1300m -XX:MaxPermSize=256m

2009-09-21 21:26:26,966 [main] ERROR kernel.plugins.dependency.AbstractKernelCon
troller : Error installing to Real: name=vfsfile:/D:/Temp/jboss-5.1.0.GA/server
/myasconfig/deploy/my-app.ear/ state=PreReal mode=Manual requiredState=Real

org.jboss.deployers.spi.DeploymentException: Error deploying prod-app.ear:
null
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:196)
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:99)
at org.jboss.deployers.vfs.spi.deployer.AbstractVFSRealDeployer.internal
Deploy(AbstractVFSRealDeployer.java:45)
at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(
AbstractRealDeployer.java:50)
at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(Deployer
Wrapper.java:171)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(Deployer
sImpl.java:1439)
at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFi
rst(DeployersImpl.java:1157)
at org.jboss.deployers.plugins.deployers.DeployersImpl.install(Deployers
Impl.java:1098)
at org.jboss.dependency.plugins.AbstractControllerContext.install(Abstra
ctControllerContext.java:348)
at org.jboss.dependency.plugins.AbstractController.install(AbstractContr
oller.java:1631)
at org.jboss.dependency.plugins.AbstractController.incrementState(Abstra
ctController.java:934)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
actController.java:1082)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
actController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
ller.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
ller.java:553)
at org.jboss.deployers.plugins.deployers.DeployersImpl.process(Deployers
Impl.java:781)
at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeploye
rImpl.java:702)
at org.jboss.system.server.profileservice.repository.MainDeployerAdapter
.process(MainDeployerAdapter.java:117)
at org.jboss.system.server.profileservice.repository.ProfileDeployAction
.install(ProfileDeployAction.java:70)
at org.jboss.system.server.profileservice.repository.AbstractProfileActi
on.install(AbstractProfileAction.java:53)
at org.jboss.system.server.profileservice.repository.AbstractProfileServ
ice.install(AbstractProfileService.java:361)
at org.jboss.dependency.plugins.AbstractControllerContext.install(Abstra
ctControllerContext.java:348)
at org.jboss.dependency.plugins.AbstractController.install(AbstractContr
oller.java:1631)
at org.jboss.dependency.plugins.AbstractController.incrementState(Abstra
ctController.java:934)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
actController.java:1082)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
actController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
ller.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
ller.java:553)
at org.jboss.system.server.profileservice.repository.AbstractProfileServ
ice.activateProfile(AbstractProfileService.java:306)
at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(
ProfileServiceBootstrap.java:271)
at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.OutOfMemoryError
at java.util.zip.Inflater.init(Native Method)
at java.util.zip.Inflater.<init>(Inflater.java:75)
at java.util.zip.ZipFile.getInflater(ZipFile.java:375)
at java.util.zip.ZipFile.getInputStream(ZipFile.java:320)
at java.util.zip.ZipFile.getInputStream(ZipFile.java:286)
at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFi
leWrapper.java:214)
at org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipE
ntryContext.java:1082)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipE
ntryHandler.java:153)
at org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:230)
at org.jboss.ejb3.Ejb3Deployment.deployUrl(Ejb3Deployment.java:650)
at org.jboss.ejb3.Ejb3Deployment.deploy(Ejb3Deployment.java:614)
at org.jboss.ejb3.Ejb3Deployment.create(Ejb3Deployment.java:491)
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:182)
... 33 more

Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1229
92 bytes for Chunk::new. Out of swap space?
Press any key to continue . . .

As you can see this happens when our ear(37M) is deployed, any idea that we can overcome this? Thanks.

[alesj]

Make sure you're running in VFS forceCopy mode:

13:52:32,843 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled.

Otherwise nested jar files are handled in-memory,
which is very memory consuming.

[yongz]

Thanks for the prompt response again Ales.

Just to confirm:

13:52:32,843 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled.

Does this message indicate we are NOT running in VFS forceCopy mode?

Make sure you're running in VFS forceCopy mode

We just use the default JBoss VFC settings, and from JBoss doc:

JBoss AS sets boss.vfs.forceCopy=true by default.

Could you please advise how I can confirm that we are running in VFS forceCopy mode

[alesj]

If you didn't change anything, then you're running in copy mode
- as indicated in JBoss docs (and your comment).
In this case you should see that log info msg in server's log/console.

[jaikiran]

Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1229 92 bytes for Chunk::new. Out of swap space?

The "CompilerThread0" reference is strange. I haven't usually seen OOM being reported against the thread name.
Just googled for this and it appears to be a bug in Sun JDK:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6487381

http://www.danielschneller.com/2007/04/exception-in-compilerthread0.html

Check if there is a new update of the JDK available and try against it.

[yongz]

Thanks Ales/JAIKIRAN

In this case you should see that log info msg in server's log/console.

The stack trace I posted are from the server console, and the server logs has the same message.

Check if there is a new update of the JDK available and try against it.

We are currently using jdk1.5.0_19, will try the latest jdk1.5.0_21.

However, it was OK when we ran jboss 4.2.2 on jdk1.5.0_16.

[yongz]

We are currently using jdk1.5.0_19, will try the latest jdk1.5.0_21.

I just did a test and same issue happened on jdk1.5.0_21.

Any suggestion that I can move forward? Thanks.

[jaikiran]

Edit the run.conf (or run.bat.conf depending on the OS you use) to add the -client to the JAVA_OPTS section:

JAVA_OPTS="-client ....//the rest of it"

and see if that works.