9 Replies Latest reply on Jan 22, 2011 9:36 PM by jnl1

    Out of Memory in AS 5.1.0.GA

    jnl1
    jnl1 Jan 20, 2011 3:35 PM

      hi...

         I have a web app the is working fine under 4.2.3.GA.   Want to migrate to  5.1.0.GA. The only code I changed was:

      org.jboss.security.Util.createPasswordHash(....)

       

      to

       

      org.jboss.crypto.CryptoUtil.createPasswordHash(....)

       

      Can someone tell me if there's a memory leak with a jar or something?   My memory params are:

       

      -server -Xms512m -Xmx512m -XX:MaxPermSize=256m  (same as in 4.2.3.GA which works fine).

       

       

      thx in advance...

       

       

      • 8823 Views
      • Tags:
        • 1. Out of Memory in AS 5.1.0.GA
          peterj
          peterj Jan 20, 2011 4:12 PM (in response to jnl1)

          JBoss AS 5.1.0 is know to use more memory than 4.2.3. I recommend increasing your heap size.

           

          Other than that, you would have to look at some heap dumps to see what is using up the space.

           

          By the way, there are several different reasons you might get a OOME. Since you didn't post the exact stack trace, I made a guess as to which OOME you got. But I could be wrong in which case my advice is useless.

            • Actions
          • 2. Out of Memory in AS 5.1.0.GA
            jnl1
            jnl1 Jan 20, 2011 4:13 PM (in response to jnl1)

            I took out the crypto information and still get the outOfMemory exception.   So, not sure why 5.1 is so different thatn 4.2.3 ??....

              • Actions
            • 3. Out of Memory in AS 5.1.0.GA
              jnl1
              jnl1 Jan 20, 2011 4:16 PM (in response to jnl1)

              thanks for the quick response.   sorry, should have included the stack trace:

               

              11:25:58,731 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space

              11:25:58,731 WARN  [HDScanner] Scan failed

              java.lang.OutOfMemoryError: Java heap space

              11:26:01,968 ERROR [STDERR]     at java.util.WeakHashMap.put(WeakHashMap.java:416)

              11:26:01,969 ERROR [STDERR]     at org.jboss.aop.AspectManager.getAdvisors(AspectManager.java:546)

              11:26:04,884 ERROR [STDERR]     at org.jboss.console.plugins.AOPLister$RefreshPoller.run(AOPLister.java:891)

              11:26:04,885 ERROR [STDERR] Exception in thread "RMI TCP Connection(idle)"

                • Actions
              • 4. Out of Memory in AS 5.1.0.GA
                peterj
                peterj Jan 20, 2011 4:20 PM (in response to jnl1)

                Hey, I guessed right!

                  • Actions
                • 5. Out of Memory in AS 5.1.0.GA
                  jnl1
                  jnl1 Jan 20, 2011 4:51 PM (in response to peterj)

                  Getting the same error with these params:

                   

                  -Xms1024m -Xmx1024m -XX:MaxPermSize=512m

                   

                  think I should try bumping it up more ?  

                    • Actions
                  • 6. Out of Memory in AS 5.1.0.GA
                    peterj
                    peterj Jan 20, 2011 5:06 PM (in response to jnl1)

                    No. You have a memory leak and you have to find out what is leaking. Try adding -XX:+HeapDumpOnOutOfMemoryError to the JAVA_OPTS. This will cause the JVM to dump the heap when it gets an OOME. You can load the dump file into VisualVM (or some similar tool) and see what it taking up the space.

                      • Actions
                    • 7. Out of Memory in AS 5.1.0.GA
                      jnl1
                      jnl1 Jan 20, 2011 8:56 PM (in response to peterj)

                      here's the stack trace.

                       

                      java.lang.OutOfMemoryError: GC overhead limit exceeded
                      Dumping heap to java_pid9460.hprof ...
                      Heap dump file created [517770450 bytes in 5.585 secs]
                      17:16:47,120 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:823)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:191)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:684)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
                              at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
                              at org.jboss.deployers.vfs.spi.structure.modified.UpdateDeleteVisitor.doVisit(UpdateDeleteVisitor.java:53)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
                              at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
                      or.java:62)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.VFS.visit(VFS.java:421)
                              at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
                      rapperModificationChecker.java:90)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:195)
                      17:16:47,120 ERROR [STDERR] Exception in thread "ZipFile Lock Reaper"
                      17:16:47,132 ERROR [STDERR] java.lang.OutOfMemoryError: GC overhead limit exceeded
                      17:16:47,132 ERROR [STDERR]     at java.util.concurrent.ConcurrentLinkedQueue.iterator(ConcurrentLinkedQueue.java:452)
                      17:16:47,133 ERROR [STDERR]     at org.jboss.virtual.plugins.context.zip.ZipFileLockReaper$ReaperTimerTask.run(ZipFileLo
                      ckReaper.java:158)
                      17:16:47,133 ERROR [STDERR]     at java.util.TimerThread.mainLoop(Timer.java:512)
                      17:16:47,133 ERROR [STDERR]     at java.util.TimerThread.run(Timer.java:462)
                      17:17:02,741 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                              at java.lang.String.substring(String.java:1939)
                              at org.jboss.virtual.plugins.vfs.helpers.PathTokenizer.getTokens(PathTokenizer.java:135)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:650)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
                              at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
                              at org.jboss.deployers.vfs.spi.structure.modified.AddVisitor.doVisit(AddVisitor.java:53)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
                              at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
                      or.java:62)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.VFS.visit(VFS.java:421)
                              at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
                      rapperModificationChecker.java:93)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:195)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:138)
                              at org.jboss.system.server.profileservice.repository.HotDeploymentRepository.getModifiedDeployments(HotDeploymen
                      tRepository.java:120)
                              at org.jboss.system.server.profile.repository.AbstractProfile.getModifiedDeployments(AbstractProfile.java:128)
                      17:17:08,171 WARN  [ClassLoaderManager] Unexpected error during load of:javax.servlet.jsp.jstl.fmt.LocalizationContext
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
                              at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
                              at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)

                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
                              at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                              at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                              at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
                              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                              at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
                              at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
                              at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
                              at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
                      17:17:12,571 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
                              at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
                              at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)

                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
                              at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                              at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                              at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
                              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                              at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
                              at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
                              at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
                              at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
                      17:17:16,686 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                      17:17:20,986 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
                      java.lang.OutOfMemoryError: Java heap space
                      17:17:20,987 ERROR [[localhost]] Exception Processing ErrorPage[exceptionType=java.lang.Throwable, location=/protected/e
                      rror.jsp]
                      javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
                              at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:257)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
                              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                              at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
                              at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:446)
                              at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
                              at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
                              at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:416)
                              at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:270)
                              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
                              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
                              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
                              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
                              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
                              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                              at java.lang.Thread.run(Thread.java:619)
                      Caused by: java.lang.OutOfMemoryError: Java heap space

                       

                       

                      thanks

                        • Actions
                      • 8. Out of Memory in AS 5.1.0.GA
                        peterj
                        peterj Jan 21, 2011 11:25 AM (in response to jnl1)

                        Ahah! The plot thickens! A "GC overhead limit exceeded" error happens because the GC is not cleaning up enough objects from the heap. There are two possible causes I can think of. One is that there is a memory leak. Actually, more than a memory leak - for some resons the code is hanging on to every object it is creating and never releasing anything. The other is that the GC settings are such that the GC finds it impossible to do its job.

                         

                        The second is easier to fix, if that is the case. So what are your GC settings? Posting the full set of JVM options would help (they are displayed right when you first start up the app server).

                         

                        As far as the memory leak goes, have you loaded java_pid9460.hprof into VisualVM? What did you see? If you haven't, be warned that you need about 3 to 4 times as much heap space when running Visual VM - since your heap was set to 512MB, you need to set the heap for VisualVM to around 1.5GB to 2GB, which means you might need to run with a 64-bit JVM.

                          • Actions
                        • 9. Out of Memory in AS 5.1.0.GA
                          jnl1
                          jnl1 Jan 22, 2011 9:36 PM (in response to peterj)

                          Thanks for the followup.   I'm running with a 64 bit VM, so will run in VisualVM when I get a chance.  But, looking at the stack, it looked like it could be related to the display tag lib, which is on the page that hangs (not all pages hang with out of memory).   So, I stripped the display tab table (after looking at the src code in display tag) and it looks like if I change this it works:

                           

                          <display:column align='left' width='20%' titleKey='document.expiration.label'>

                           

                          to

                           

                          <display:column align='left' width='20%' title='Expiration Date'>

                           

                          Notice the title, titleKey element.   For some reason, display tag doesn't like the ref to the properties file.   so, going to research that some more.   will post my results when done...

                           

                          thanks

                            • Actions