1 2 Previous Next 24 Replies Latest reply on Jan 30, 2013 5:19 PM by Ramesh Pitani

    OutOfMemoryError in JBoss 5.01

    Yong Zhang Newbie

      Hi,

      Env:
      * jdk1.5.0_19
      * jboss-5.0.1.GA

      We just upgraded our production server to jboss-5.0.1.GA from jboss-4.2.2.GA today, and it caused huge problem on production:

      With JVM settings:

      1. -Xms1300m -Xmx1300m -XX:MaxPermSize=256m

      jboss-5.0.1.GA gives OOM during startup:
      Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 32776 bytes for Chunk::new. Out of swap space?

      While this settings works perfectly in jboss-4.2.2.GA.

      2. -Xms1200m -Xmx1200m -XX:MaxPermSize=256m

      jboss-5.0.1.GA starts OK, but server got OOM again when the load increased and users started to login and use the server.

      3. -Xms1024m -Xm1024m -XX:MaxPermSize=256m

      jboss-5.0.1.GA starts OK, but server still got OOM again when the load increased and users started to login and use the server.

      3. -Xms768m -Xm768m -XX:MaxPermSize=256m

      For now we mitigate the issue with the above memory settings, but we cannot stay with this heap size for long.

      Does anyone have this issue before in jboss-5.0.1.GA?

      We believe boss-5.0.1.GA has undergone rigorous testing, including load testing, and should not have an issue like this.

      However, I have an impression that jboss-5.0.1.GA uses too much non-heap memory which prevent it from creating more threads under load. Something to do with the loading of libs in jboss-5.0.1.GA?

      Thanks in advance.

        • 1. Re: OutOfMemoryError in JBoss 5.01
          Ales Justin Master

          There is this memory leak:
          - https://jira.jboss.org/jira/browse/JBCL-119
          but if you're not re-deploying like mad +
          have explicit CL dependencies on "static" deployment,
          you're not hitting this.

          • 2. Re: OutOfMemoryError in JBoss 5.01
            jaikiran pai Master

            In addition to what Ales said - it's better to move to AS 5.1.0 instead of 5.0.1 because 5.1.0 has a lot of bug fixes.

            • 3. Re: OutOfMemoryError in JBoss 5.01
              Yong Zhang Newbie

              Thanks a lot for the response.

              but if you're not re-deploying like mad + have explicit CL dependencies on "static" deployment


              Our EAR is about 37 MB, and we are using JBossws-metro.

              Would you mind explaining a bit more about the re-deploying and explicit CL dependencies on "static" deployment?

              Is there a way that we can avoid this?

              • 4. Re: OutOfMemoryError in JBoss 5.01
                Ales Justin Master

                 

                "yongz" wrote:

                Would you mind explaining a bit more about the re-deploying and explicit CL dependencies on "static" deployment?

                Like I said, first, I doubt you're hitting this,
                as explicit CL dependencies are not yet common in recent deployments.
                But I might be wrong. :-)

                The scenario goes like this.
                You have deployments A and B, where B expresses explicit CL dependency on A
                (via jboss-classloading.xml or some other CLMetaData info).
                Then you re-deploy B a lot of times, meaning A's dependsOnMe info grows big ==> OOME.

                • 5. Re: OutOfMemoryError in JBoss 5.01
                  Yong Zhang Newbie

                  Thanks Ales/JAIKIRAN, much appreciated.

                  via jboss-classloading.xml or some other CLMetaData info


                  The issue you described is close to what I saw, but we did not make any of the above CL changes to JBoss or in our application.

                  A few more questions:
                  1. Could you also confirm this CL issue would cause the server cannot handle load(more threads)?
                  2. Is there something(jars, etc) we can upgrade in AS 5.01?
                  3. As JAIKIRAN suggested, will this issue be fixed in 5.1.0?

                  Sorry about so many questions, but I have to sort this out on prod quickly.



                  • 6. Re: OutOfMemoryError in JBoss 5.01
                    Ales Justin Master

                     

                    "yongz" wrote:

                    1. Could you also confirm this CL issue would cause the server cannot handle load(more threads)?

                    This is a mem-leak, so it can cause a bunch of weird issues. :-(

                    "yongz" wrote:

                    2. Is there something(jars, etc) we can upgrade in AS 5.01?

                    You could, but it's risky as you don't exactly know how things/libs are related.
                    So, the best thing to do is to use the latest version of AS.

                    "yongz" wrote:

                    3. As JAIKIRAN suggested, will this issue be fixed in 5.1.0?

                    5.1.0 is already out, whereas this was just recently found and fixed.
                    It will be part of upcoming AS release, 5.2.x.


                    • 7. Re: OutOfMemoryError in JBoss 5.01
                      Yong Zhang Newbie

                      Hi Ales,

                      Unfortunately I was able to reproduce this issue on 5.1.0 with:

                      -Xms1300m -Xmx1300m -XX:MaxPermSize=256m

                      2009-09-21 21:26:26,966 [main] ERROR kernel.plugins.dependency.AbstractKernelCon
                      troller : Error installing to Real: name=vfsfile:/D:/Temp/jboss-5.1.0.GA/server
                      /myasconfig/deploy/my-app.ear/ state=PreReal mode=Manual requiredState=Real

                      org.jboss.deployers.spi.DeploymentException: Error deploying prod-app.ear:
                      null
                      at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:196)
                      at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:99)
                      at org.jboss.deployers.vfs.spi.deployer.AbstractVFSRealDeployer.internal
                      Deploy(AbstractVFSRealDeployer.java:45)
                      at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(
                      AbstractRealDeployer.java:50)
                      at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(Deployer
                      Wrapper.java:171)
                      at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(Deployer
                      sImpl.java:1439)
                      at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFi
                      rst(DeployersImpl.java:1157)
                      at org.jboss.deployers.plugins.deployers.DeployersImpl.install(Deployers
                      Impl.java:1098)
                      at org.jboss.dependency.plugins.AbstractControllerContext.install(Abstra
                      ctControllerContext.java:348)
                      at org.jboss.dependency.plugins.AbstractController.install(AbstractContr
                      oller.java:1631)
                      at org.jboss.dependency.plugins.AbstractController.incrementState(Abstra
                      ctController.java:934)
                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
                      actController.java:1082)
                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
                      actController.java:984)
                      at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
                      ller.java:822)
                      at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
                      ller.java:553)
                      at org.jboss.deployers.plugins.deployers.DeployersImpl.process(Deployers
                      Impl.java:781)
                      at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeploye
                      rImpl.java:702)
                      at org.jboss.system.server.profileservice.repository.MainDeployerAdapter
                      .process(MainDeployerAdapter.java:117)
                      at org.jboss.system.server.profileservice.repository.ProfileDeployAction
                      .install(ProfileDeployAction.java:70)
                      at org.jboss.system.server.profileservice.repository.AbstractProfileActi
                      on.install(AbstractProfileAction.java:53)
                      at org.jboss.system.server.profileservice.repository.AbstractProfileServ
                      ice.install(AbstractProfileService.java:361)
                      at org.jboss.dependency.plugins.AbstractControllerContext.install(Abstra
                      ctControllerContext.java:348)
                      at org.jboss.dependency.plugins.AbstractController.install(AbstractContr
                      oller.java:1631)
                      at org.jboss.dependency.plugins.AbstractController.incrementState(Abstra
                      ctController.java:934)
                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
                      actController.java:1082)
                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(Abstr
                      actController.java:984)
                      at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
                      ller.java:822)
                      at org.jboss.dependency.plugins.AbstractController.change(AbstractContro
                      ller.java:553)
                      at org.jboss.system.server.profileservice.repository.AbstractProfileServ
                      ice.activateProfile(AbstractProfileService.java:306)
                      at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(
                      ProfileServiceBootstrap.java:271)
                      at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
                      461)
                      at org.jboss.Main.boot(Main.java:221)
                      at org.jboss.Main$1.run(Main.java:556)
                      at java.lang.Thread.run(Thread.java:595)
                      Caused by: java.lang.OutOfMemoryError
                      at java.util.zip.Inflater.init(Native Method)
                      at java.util.zip.Inflater.<init>(Inflater.java:75)
                      at java.util.zip.ZipFile.getInflater(ZipFile.java:375)
                      at java.util.zip.ZipFile.getInputStream(ZipFile.java:320)
                      at java.util.zip.ZipFile.getInputStream(ZipFile.java:286)
                      at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFi
                      leWrapper.java:214)
                      at org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipE
                      ntryContext.java:1082)
                      at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipE
                      ntryHandler.java:153)
                      at org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:230)
                      at org.jboss.ejb3.Ejb3Deployment.deployUrl(Ejb3Deployment.java:650)
                      at org.jboss.ejb3.Ejb3Deployment.deploy(Ejb3Deployment.java:614)
                      at org.jboss.ejb3.Ejb3Deployment.create(Ejb3Deployment.java:491)
                      at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:182)
                      ... 33 more

                      Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1229
                      92 bytes for Chunk::new. Out of swap space?
                      Press any key to continue . . .


                      As you can see this happens when our ear(37M) is deployed, any idea that we can overcome this? Thanks.

                      • 8. Re: OutOfMemoryError in JBoss 5.01
                        Ales Justin Master

                        Make sure you're running in VFS forceCopy mode:

                        13:52:32,843 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled.
                        

                        Otherwise nested jar files are handled in-memory,
                        which is very memory consuming.

                        • 9. Re: OutOfMemoryError in JBoss 5.01
                          Yong Zhang Newbie

                          Thanks for the prompt response again Ales.

                          Just to confirm:

                          13:52:32,843 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled.


                          Does this message indicate we are NOT running in VFS forceCopy mode?

                          Make sure you're running in VFS forceCopy mode


                          We just use the default JBoss VFC settings, and from JBoss doc:

                          JBoss AS sets boss.vfs.forceCopy=true by default.


                          Could you please advise how I can confirm that we are running in VFS forceCopy mode

                          • 10. Re: OutOfMemoryError in JBoss 5.01
                            Ales Justin Master

                            If you didn't change anything, then you're running in copy mode
                            - as indicated in JBoss docs (and your comment).
                            In this case you should see that log info msg in server's log/console.

                            • 11. Re: OutOfMemoryError in JBoss 5.01
                              jaikiran pai Master

                               

                              Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 1229 92 bytes for Chunk::new. Out of swap space?


                              The "CompilerThread0" reference is strange. I haven't usually seen OOM being reported against the thread name.
                              Just googled for this and it appears to be a bug in Sun JDK:

                              http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6487381

                              http://www.danielschneller.com/2007/04/exception-in-compilerthread0.html

                              Check if there is a new update of the JDK available and try against it.

                              • 12. Re: OutOfMemoryError in JBoss 5.01
                                Yong Zhang Newbie

                                Thanks Ales/JAIKIRAN

                                In this case you should see that log info msg in server's log/console.


                                The stack trace I posted are from the server console, and the server logs has the same message.

                                Check if there is a new update of the JDK available and try against it.


                                We are currently using jdk1.5.0_19, will try the latest jdk1.5.0_21.

                                However, it was OK when we ran jboss 4.2.2 on jdk1.5.0_16.

                                • 13. Re: OutOfMemoryError in JBoss 5.01
                                  Yong Zhang Newbie

                                   

                                  We are currently using jdk1.5.0_19, will try the latest jdk1.5.0_21.


                                  I just did a test and same issue happened on jdk1.5.0_21.

                                  Any suggestion that I can move forward? Thanks.

                                  • 14. Re: OutOfMemoryError in JBoss 5.01
                                    jaikiran pai Master

                                    Edit the run.conf (or run.bat.conf depending on the OS you use) to add the -client to the JAVA_OPTS section:

                                    JAVA_OPTS="-client ....//the rest of it"


                                    and see if that works.


                                    1 2 Previous Next