12 Replies Latest reply on Aug 11, 2011 9:19 PM by amathewjboss1

    Out of Memory Error with Jboss 5.1.0

    amathewjboss1

      Hi,

       

         We have been testing Jboss 5.1 for the last 6 months and as a final test before the production we are doing some load testing. During our load test we are getting OutOfMemoryError and finally Jboss will crash. Below is error we are getting:

       

      Caused by: java.lang.OutOfMemoryError

              at java.util.zip.ZipFile.open(Native Method)

              at java.util.zip.ZipFile.<init>(ZipFile.java:114)

              at java.util.zip.ZipFile.<init>(ZipFile.java:131)

              at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.ensureZipFile(ZipFileWrapper.java:175)

              at org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFileWrapper.java:213)

              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipEntryContext.java:1082)

              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipEntryHandler.java:153)

              at org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:230)

              at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getResourceAsStream(VFSClassLoaderPolicy.java:483)

              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:534)

              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)

              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)

              at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)

              at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)

              at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)

              at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)

              at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)

       

           We are running the load test in a clustered environment with 3 app servers in one cluster (and with 4 web servers). Our environment is Linux with 4GB Memory and with JDK 1.6.05

          

          The JAVA_OPTS settings are:

       

        JAVA_OPTS="-server -verbose:gc -Xms920M -Xmx1850M"

        JAVA_OPTS="${JAVA_OPTS} -XX:MaxPermSize=128M"

       

           I read many conversations in this forum regarding out of memory error. One of the main posting  which matched the above error was http://community.jboss.org/thread/160201?start=15&tstart=0   (https://issues.jboss.org/browse/JBVFS-134). Can you please whether is a real bug in JBoss 5.1? Also does this already fixed in JBoss 5.1? We are all set to upgrade the production environment with JBoss 5.1 in another 2 weeks and we basically stumped now after seeing these errors during our load testing. Please help.

       

      Thanks

      Anil Mathew

        • 1. Re: Out of Memory Error with Jboss 5.1.0
          amathewjboss1

          I also wanted to mention that i applied the change which is suggested by Jaikiran in this post http://community.jboss.org/message/555231#555231, but no luck and still getting the above outof memory error which i mentioned in the first posting.

           

          I also tried with various JAVA_OPTS setting - like setting to MaxPermSize to 256M.

           

          Can some one please help me?

           

          • 2. Re: Out of Memory Error with Jboss 5.1.0
            peterj

            Based on the first item in the stack trace:

             

            Caused by: java.lang.OutOfMemoryError

                    at java.util.zip.ZipFile.open(Native Method)

             

            This is not a heap size issue nor a permgen size issue. Instead, it is a native method that cannot get more memory space. Therefore you are running out of either physical memory on the box, or the JVM has exhasted its address space.

             

            Is this running on a 32-bit OS with a 32-bit JVM? If so, try reducing the -Xmz size - that should free up more of the JVM's address space for use by the native methods.

            • 3. Re: Out of Memory Error with Jboss 5.1.0
              amathewjboss1

              Thank You Peter for the response (very much appreciated).

               

              I spend many hours analyzing the heap dump and I do agree to your point that this is not a heap size nor a permgen size issue. That means it is a native method memory issue. But what i don't understand here is why native methods is asking for so much memory in Jboss 5.1.

               

              i.e when i run the same load test against JBoss 4.0.3 version I don't see any issue with the same -Xmx1850M value. This makes me think that we have some memory leakage in Jboss 5.1. I was wondering this memory leakage issue is the one mentioned in this URL - https://issues.jboss.org/browse/JBVFS-134. If so can I safely use the jboss-vfs.jar which is on this issue? In fact i download the Jboss 5.1 package an year ago.

               

              Thanks

              Anil Mathew

              • 4. Re: Out of Memory Error with Jboss 5.1.0
                peterj

                Sorry, but I can't answer the question as to why 5.1.0 would use more native memory, and while I suspect that the issue does have something to do with VFS, I don't know if the JIRA issue you pointed out has anything to do with what you are experiencing. Maybe someone on the JBoss team who knows VFS internals could offer some insight.

                • 5. Re: Out of Memory Error with Jboss 5.1.0
                  amathewjboss1

                  Thanks Again Peter for your time and for your thoughts.

                   

                  I hope i will get some insight from Jboss team after reading this post. As i mentioned earlier all test were going good for the past 6 - 8 months and now it seems like I am getting disappointed with the way JBoss 5.1 acting for this load test by spitting out Out of Memory error. In fact after a while the JBoss crashes (of course). Again insight from the Jboss team will be highly helpful for me to put this JBoss 5.1 to production.

                   

                  Thanks Again

                  Anil Mathew

                  • 6. Re: Out of Memory Error with Jboss 5.1.0
                    jaikiran
                    • 7. Re: Out of Memory Error with Jboss 5.1.0
                      jaikiran
                      • 8. Re: Out of Memory Error with Jboss 5.1.0
                        amathewjboss1

                        Thank You Jaikiran for your thoughts.

                         

                        The URL in your last reply (http://community.jboss.org/thread/160201?start=15&tstart=0) is the exact one I was pointing to in my first posting. In this http://community.jboss.org/thread/160201?start=15&tstart=0, they mentioned about https://jira.jboss.org/browse/JBVFS-134 with a solution of attachment of jboss-vfs.jar. Will you be able to tell me whether I can replace my current jboss-vfs.jar (which came with JBoss 5.1 download) with the attachment in the https://jira.jboss.org/browse/JBVFS-134? The reason i asked this question is from all the three posting which you send, it seems it this is the JBoss fix to get over with the JDK issue which is reported in

                        http://bugs.sun.com/view_bug.do?bug_id=6735255

                         

                        Thanks

                        Anil Mathew

                         

                         

                         

                        • 9. Re: Out of Memory Error with Jboss 5.1.0
                          jaikiran

                          Anil Mathew wrote:

                           

                          Will you be able to tell me whether I can replace my current jboss-vfs.jar (which came with JBoss 5.1 download) with the attachment in the https://jira.jboss.org/browse/JBVFS-134?

                          Sorry, I haven't tried it myself, so I can't say whether it fixes the issue and whether or not it is safe to switch to that new version in JBoss AS 5.1.

                          • 10. Re: Out of Memory Error with Jboss 5.1.0
                            jaikiran

                            By the way, did you try it by placing this new jar in AS 5.1 and see if it fixes the issue?

                            • 11. Re: Out of Memory Error with Jboss 5.1.0
                              amathewjboss1

                              I didn't tried it yet. I did change the setup which you mentioned in http://community.jboss.org/message/555231#555231 and I am not sure whether that helped or not. But we did a fresh deployment of our application and ran LOAD test for 2 days and now we can't reproduce the memory issues which i mentioned initially.

                               

                              If we are able to reproduce it again, i will go ahead and replace the jboss-vfs.jar and see how that helps.

                               

                              Also i am trying to find out the JBoss configuration file where i can configure number of concurrent threads in Jboss and I would like to lower it to about the number of connections. i.e it makes no sense having (substantially?) more threads than connections. Can you point me to this configuration file please?

                               

                              Thanks for all the support.

                              Anil Mathew

                              • 12. Re: Out of Memory Error with Jboss 5.1.0
                                amathewjboss1

                                I just want to give you a update. After updating /opt/pvo/cots/jboss/server/all.pvo/deploy/jbossweb.sar/server.xml with approriate maxThreads (depends on the connections), we were able to fix this issue.

                                 

                                      <Connector port="7001" address="${jboss.bind.address}"
                                         maxThreads="250" strategy="ms" maxHttpHeaderSize="8192"
                                         emptySessionPath="true"
                                         enableLookups="false" redirectPort="7002" acceptCount="100"
                                         connectionTimeout="20000" disableUploadTimeout="true"/>

                                 

                                      <!-- SSL/TLS Connector configuration using the admin devl guide keystore -->
                                      <Connector protocol="HTTP/1.1" SSLEnabled="true" port="7002" address="${jboss.bind.address}"
                                           maxThreads="100" strategy="ms" maxHttpHeaderSize="8192"
                                           emptySessionPath="true"
                                           scheme="https" secure="true" clientAuth="false"
                                           keystoreFile="${jboss.server.home.dir}/conf/chap8.keystore"
                                           keystorePass="rmi+ssl" sslProtocol = "TLS" />

                                I DID not replaced the jboss-vfs.jar so far and all looks good. Thanks again.

                                 

                                Anil Mathew