10 Replies Latest reply on May 19, 2013 11:08 PM by bondchan921

    java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file

    pitani

      I am getting in to weird problem when I am trying to upgrade JBoss to 5.1.0-GA. (OutOfMemory and OutOfMemory with Native threads). I am having ear file which is having several libraries inside and I am using jdk-1.6.0_29(32b). I am getting this error in the JBoss-5.1.0 start up itself. If you are aware of this problem, Can you please help me here and give your suggestions about this problem? What is the best solution here?

      1. If I start server with –Xmx2048m and –Xms2048m, I am getting in to above problems. If I reduce memory to –Xmx1536m, then this problem go away. I am assuming that JBoss5 is not getting enough native memory for loading jar files.
      2. I tried to update jboss-vfs.jar file to 2.2.1 version. Still having same error. Should I upgrade jboss-vfs.jar to higher than this version ex: 3.1.0?
      3. If I upgrade jdk to 1.7. Does it resolve this problem?
      4. I have added –Djboss.vfs.forVfsJar=true to JBoss start up script. Then server started successfully. Is it safe to use this parmeter. We want to know what is pros and cons of using this parameter.

      2013-01-09 05:18:16,010 DEBUG [org.jboss.bootstrap.microcontainer.ServerImpl] Installing life thread Thread[JBossLifeThread,5,jboss]
      2013-01-09 05:18:16,011 DEBUG [org.jboss.bootstrap.microcontainer.ServerImpl] Failed to start
      java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:640)
      at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:476)
      at org.jboss.Main.boot(Main.java:221)
      at org.jboss.Main$1.run(Main.java:556)
      at java.lang.Thread.run(Thread.java:662)

      2013-01-09 05:18:18,977 ERROR [org.springframework.batch.core.step.AbstractStep] Encountered an error executing the step
      java.lang.OutOfMemoryError
      at java.util.zip.ZipFile.open(Native Method)
      at java.util.zip.ZipFile.<init>(ZipFile.java:127)
      at java.util.zip.ZipFile.<init>(ZipFile.java:143)
      at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.ensureZipFile(ZipFileWrapper.java:175)
      at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFileWrapper.java:213)
      at org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipEntryContext.java:1082)
      at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipEntryHandler.java:153)
      at org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:230)
      at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getResourceAsStream(VFSClassLoaderPolicy.java:483)
      at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)
      at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
      at java.security.AccessController.doPrivileged(Native Method)
      at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
      at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:789)
      at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at org.springframework.batch.classify.SubclassClassifier$ClassComparator.<init>(SubclassClassifier.java:135)
      at org.springframework.batch.classify.SubclassClassifier.classify(SubclassClassifier.java:109)
      at org.springframework.batch.core.step.item.SimpleRetryExceptionHandler.handleException(SimpleRetryExceptionHandler.java:80)
      at org.springframework.batch.repeat.support.RepeatTemplate.doHandle(RepeatTemplate.java:293)
      at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:219)
      at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
      at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:248)

        • 1. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
          alesj


          2. I tried to update jboss-vfs.jar file to 2.2.1 version. Still having same error. Should I upgrade jboss-vfs.jar to higher than this version ex: 3.1.0?

          AS5 can only run on VFS v2, v3 is for AS6 and higher.

           

          3. If I upgrade jdk to 1.7. Does it resolve this problem?

          I doubt it changes anything.

           

          4. I have added –Djboss.vfs.forVfsJar=true to JBoss start up script. Then server started successfully. Is it safe to use this parmeter. We want to know what is pros and cons of using this parameter.

          This means you're using old jar handling code, writen in VFS v1.

          If it works for you, fine, but there were some issues, hence we re-wrote it.

           

          Search the forums for other similar issue.

          e.g. it might be that you're not caching VFS roots properly,

          hence we need to re-create / re-unpackage nested jar every time

          • 2. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
            pitani

            Thank you very much for your reply.

             

            Initially we have this problem(tmp folder in JBoss is filling quickly). Then we changed cache implementation from CombinedVFSCache to LRUVFSCache in vfs.xml.

             

            Do you think this vfs.xml is causing above problem? If we change to vfs.xml IterableTimedVFSCache, does it solve all problems?

            https://issues.jboss.org/browse/JBAS-7628

            • 3. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
              pitani

              We are getting Native Memory issues only with application require 2GB heap. I read below point in JBoss-5.1.0 docs. we are using Java 1.6.0_29(32b). Do you think should we use JBoss with  JDK 1.6.0_29(64 bit)?

               

              Applications that run with more than 1.5 GB of RAM (including free space for garbage collection optimization) should utilize the 64-bit JVM.

              • 4. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                alesj

                Do you have all permanent VFS roots covered in vfs.xml?

                • 5. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                  pitani

                  Hi Ales, I dont have permanent roots in my vfs.xml. Do you want me to add permanent roots to below vfs.xml and try this fix?

                   

                  <?xml version="1.0" encoding="UTF-8"?>

                  <!--

                      The JBossAS initializer configuration.

                  -->

                  <deployment xmlns="urn:jboss:bean-deployer:2.0">

                          <bean name="VFSCache">

                                  <constructor factoryClass="org.jboss.virtual.spi.cache.VFSCacheFactory"

                                          factoryMethod="getInstance">

                                          <!-- Use the LRUVFSCache implementation -->

                                          <parameter>org.jboss.virtual.plugins.cache.LRUVFSCache</parameter>

                                          <parameter>

                                                  <map keyClass="java.lang.String" valueClass="java.lang.String">

                                                          <entry>

                                                                  <key>jboss.vfs.cache.LRUPolicyCaching.min</key>

                                                                  <value>2</value>

                                                          </entry>

                                                          <entry>

                                                                  <key>jboss.vfs.cache.LRUPolicyCaching.max</key>

                                                                  <value>15</value>

                                                          </entry>

                                                  </map>

                                          </parameter>

                                  </constructor>

                                  <start ignored="true" />

                          </bean>

                   

                   

                          <bean name="VfsNamesExceptionHandler"

                                  class="org.jboss.virtual.plugins.context.helpers.NamesExceptionHandler">

                                  <constructor>

                                          <parameter class="java.lang.String">sqljdbc.jar</parameter>

                                  </constructor>

                          </bean>

                                      <bean name="JBossVFSInitializer"

                                  class="org.jboss.virtual.plugins.cache.PreInitializeVFSContexts">

                                  <property name="initializedVFSContexts" class="java.util.Map">

                                          <map keyClass="java.net.URL" valueClass="org.jboss.virtual.spi.ExceptionHandler">

                                                  <entry>

                                                          <key>${jboss.lib.url}</key>

                                                          <value>

                                                                  <null />

                                                          </value>

                                                  </entry>

                                                  <entry>

                                                          <key>${jboss.common.lib.url}</key>

                                                          <value>

                                                                  <inject bean="VfsNamesExceptionHandler" />

                                                          </value>

                                                  </entry>

                                                  <entry>

                                                          <key>${jboss.server.lib.url}</key>

                                                          <value>

                                                                  <inject bean="VfsNamesExceptionHandler" />

                                                          </value>

                                                  </entry>

                                                  <entry>

                                                          <key>${jboss.server.home.url}deploy</key>

                                                          <value>

                                                                  <inject bean="VfsNamesExceptionHandler" />

                                                          </value>

                                                  </entry>

                                          </map>

                                  </property>

                                  <property name="holdReference">true</property>

                          </bean>

                  </deployment>

                  • 6. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                    alesj

                    This entry keys are permanent roots -- check if your usage / deployments fall under any of these.

                    (btw: there has been few discussions on this subject)

                                        <bean name="JBossVFSInitializer"

                                    class="org.jboss.virtual.plugins.cache.PreInitializeVFSContexts">

                                    <property name="initializedVFSContexts" class="java.util.Map">

                                            <map keyClass="java.net.URL" valueClass="org.jboss.virtual.spi.ExceptionHandler">

                                                    <entry>

                                                            <key>${jboss.lib.url}</key>

                                                            <value>

                                                                    <null />

                                                            </value>

                                                    </entry>

                                                    <entry>

                                                            <key>${jboss.common.lib.url}</key>

                                                            <value>

                                                                    <inject bean="VfsNamesExceptionHandler" />

                                                            </value>

                                                    </entry>

                                                    <entry>

                                                            <key>${jboss.server.lib.url}</key>

                                                            <value>

                                                                    <inject bean="VfsNamesExceptionHandler" />

                                                            </value>

                                                    </entry>

                                                    <entry>

                                                            <key>${jboss.server.home.url}deploy</key>

                                                            <value>

                                                                    <inject bean="VfsNamesExceptionHandler" />

                                                            </value>

                                                    </entry>

                                            </map>

                                    </property>

                                    <property name="holdReference">true</property>

                            </bean>

                    • 7. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                      pitani

                      Ales, JBoss-5.1.0 has CombinedVFSCache in vfs.xml default. When we got disk space problems, we have updated vfs.xml to LRUVFSCache based on some forums. Can you please guide us which Cache should use? How to decide this?

                      • 8. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                        alesj

                        Combined cache is combination of two caches:

                        * permanent -- for static, aka known, deployment roots; deployers/, deploy/, lib/, etc

                        * temp -- for potential non-permanent roots

                         

                        It could be that your permanent stuff is just really big.

                         

                        btw: AS6 (pretty similar to AS5) and AS7 use VFS3, which is more lightweight

                        • 9. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                          pitani

                          Thank you very mcuh for helping me. Yes. My ear file is so big around 48MB. What I have done to my application is

                           

                          1. With Original jboss-vfs.jar(2.1.2) I am able to reproduce OOM error. I have tested my application JBoss-vfs 2.2.1.GA. Still I am reproducing same issue in my application (java.lang.OutOfMemoryError: unable to create new native thread).

                           

                          What I have done is I downloaded source code of jboss-vfs-2.1.2 and then applied patch files i.e. change.tar.gz from https://issues.jboss.org/browse/JBVFS-159. After I've applied this patch, I have started my server multiple times. I am not getting OOM error(java.lang.OutOfMemoryError: unable to create new native thread) gone.

                           

                          Can you please tell me whether I am going in right direction for OOM Fix?

                           

                          2. And also I have applied patch (https://issues.jboss.org/browse/JBVFS-120) because my server is taking 4 minutes to start up.

                          • 10. Re: java.lang.OutOfMemoryError during JBoss-5.1.0 startup with my ear file
                            bondchan921

                            Hi Ales,

                             

                            For upgrade jboss-vfs.jar file to 2.2.1 version, is there any other jars having dependency and need upgrade as well ?

                             

                             

                            Thanks.