9 Replies Latest reply on Jan 22, 2011 9:36 PM by Jon Loe

    Out of Memory in AS 5.1.0.GA

    Jon Loe Newbie

      hi...

         I have a web app the is working fine under 4.2.3.GA.   Want to migrate to  5.1.0.GA. The only code I changed was:

      org.jboss.security.Util.createPasswordHash(....)

       

      to

       

      org.jboss.crypto.CryptoUtil.createPasswordHash(....)

       

      Can someone tell me if there's a memory leak with a jar or something?   My memory params are:

       

      -server -Xms512m -Xmx512m -XX:MaxPermSize=256m  (same as in 4.2.3.GA which works fine).

       

       

      thx in advance...

       

       

        • 1. Out of Memory in AS 5.1.0.GA
          Peter Johnson Master

          JBoss AS 5.1.0 is know to use more memory than 4.2.3. I recommend increasing your heap size.

           

          Other than that, you would have to look at some heap dumps to see what is using up the space.

           

          By the way, there are several different reasons you might get a OOME. Since you didn't post the exact stack trace, I made a guess as to which OOME you got. But I could be wrong in which case my advice is useless.

          • 2. Out of Memory in AS 5.1.0.GA
            Jon Loe Newbie

            I took out the crypto information and still get the outOfMemory exception.   So, not sure why 5.1 is so different thatn 4.2.3 ??....

            • 3. Out of Memory in AS 5.1.0.GA
              Jon Loe Newbie

              thanks for the quick response.   sorry, should have included the stack trace:

               

              11:25:58,731 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space

              11:25:58,731 WARN  [HDScanner] Scan failed

              java.lang.OutOfMemoryError: Java heap space

              11:26:01,968 ERROR [STDERR]     at java.util.WeakHashMap.put(WeakHashMap.java:416)

              11:26:01,969 ERROR [STDERR]     at org.jboss.aop.AspectManager.getAdvisors(AspectManager.java:546)

              11:26:04,884 ERROR [STDERR]     at org.jboss.console.plugins.AOPLister$RefreshPoller.run(AOPLister.java:891)

              11:26:04,885 ERROR [STDERR] Exception in thread "RMI TCP Connection(idle)"

              • 4. Out of Memory in AS 5.1.0.GA
                Peter Johnson Master

                Hey, I guessed right!

                • 5. Out of Memory in AS 5.1.0.GA
                  Jon Loe Newbie

                  Getting the same error with these params:

                   

                  -Xms1024m -Xmx1024m -XX:MaxPermSize=512m

                   

                  think I should try bumping it up more ?  

                  • 6. Out of Memory in AS 5.1.0.GA
                    Peter Johnson Master

                    No. You have a memory leak and you have to find out what is leaking. Try adding -XX:+HeapDumpOnOutOfMemoryError to the JAVA_OPTS. This will cause the JVM to dump the heap when it gets an OOME. You can load the dump file into VisualVM (or some similar tool) and see what it taking up the space.

                    • 7. Out of Memory in AS 5.1.0.GA
                      Jon Loe Newbie

                      here's the stack trace.

                       

                      java.lang.OutOfMemoryError: GC overhead limit exceeded
                      Dumping heap to java_pid9460.hprof ...
                      Heap dump file created [517770450 bytes in 5.585 secs]
                      17:16:47,120 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:823)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:191)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:684)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
                              at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
                              at org.jboss.deployers.vfs.spi.structure.modified.UpdateDeleteVisitor.doVisit(UpdateDeleteVisitor.java:53)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
                              at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
                      or.java:62)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.VFS.visit(VFS.java:421)
                              at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
                      rapperModificationChecker.java:90)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:195)
                      17:16:47,120 ERROR [STDERR] Exception in thread "ZipFile Lock Reaper"
                      17:16:47,132 ERROR [STDERR] java.lang.OutOfMemoryError: GC overhead limit exceeded
                      17:16:47,132 ERROR [STDERR]     at java.util.concurrent.ConcurrentLinkedQueue.iterator(ConcurrentLinkedQueue.java:452)
                      17:16:47,133 ERROR [STDERR]     at org.jboss.virtual.plugins.context.zip.ZipFileLockReaper$ReaperTimerTask.run(ZipFileLo
                      ckReaper.java:158)
                      17:16:47,133 ERROR [STDERR]     at java.util.TimerThread.mainLoop(Timer.java:512)
                      17:16:47,133 ERROR [STDERR]     at java.util.TimerThread.run(Timer.java:462)
                      17:17:02,741 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                              at java.lang.String.substring(String.java:1939)
                              at org.jboss.virtual.plugins.vfs.helpers.PathTokenizer.getTokens(PathTokenizer.java:135)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:650)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
                              at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
                      ava:689)
                              at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
                              at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
                              at org.jboss.deployers.vfs.spi.structure.modified.AddVisitor.doVisit(AddVisitor.java:53)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
                              at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
                      or.java:62)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
                              at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
                              at org.jboss.virtual.VFS.visit(VFS.java:421)
                              at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
                              at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
                      rapperModificationChecker.java:93)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:195)
                              at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
                      AbstractStructureModificationChecker.java:138)
                              at org.jboss.system.server.profileservice.repository.HotDeploymentRepository.getModifiedDeployments(HotDeploymen
                      tRepository.java:120)
                              at org.jboss.system.server.profile.repository.AbstractProfile.getModifiedDeployments(AbstractProfile.java:128)
                      17:17:08,171 WARN  [ClassLoaderManager] Unexpected error during load of:javax.servlet.jsp.jstl.fmt.LocalizationContext
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
                              at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
                              at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)

                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
                              at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                              at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                              at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
                              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                              at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
                              at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
                              at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
                              at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
                      17:17:12,571 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
                      java.lang.OutOfMemoryError: Java heap space
                              at java.util.Arrays.copyOf(Arrays.java:2882)
                              at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
                              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
                              at java.lang.StringBuilder.append(StringBuilder.java:119)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
                              at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
                              at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
                              at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
                              at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)

                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
                              at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
                              at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
                              at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
                              at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
                              at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
                              at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
                              at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
                              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                              at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
                              at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
                              at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
                              at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
                              at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
                      17:17:16,686 WARN  [HDScanner] Scan failed
                      java.lang.OutOfMemoryError: Java heap space
                      17:17:20,986 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
                      java.lang.OutOfMemoryError: Java heap space
                      17:17:20,987 ERROR [[localhost]] Exception Processing ErrorPage[exceptionType=java.lang.Throwable, location=/protected/e
                      rror.jsp]
                      javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
                              at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:257)
                              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
                              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
                              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                              at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
                              at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:446)
                              at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
                              at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
                              at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:416)
                              at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:270)
                              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
                              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                              at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
                              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
                              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
                              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
                              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
                              at java.lang.Thread.run(Thread.java:619)
                      Caused by: java.lang.OutOfMemoryError: Java heap space

                       

                       

                      thanks

                      • 8. Out of Memory in AS 5.1.0.GA
                        Peter Johnson Master

                        Ahah! The plot thickens! A "GC overhead limit exceeded" error happens because the GC is not cleaning up enough objects from the heap. There are two possible causes I can think of. One is that there is a memory leak. Actually, more than a memory leak - for some resons the code is hanging on to every object it is creating and never releasing anything. The other is that the GC settings are such that the GC finds it impossible to do its job.

                         

                        The second is easier to fix, if that is the case. So what are your GC settings? Posting the full set of JVM options would help (they are displayed right when you first start up the app server).

                         

                        As far as the memory leak goes, have you loaded java_pid9460.hprof into VisualVM? What did you see? If you haven't, be warned that you need about 3 to 4 times as much heap space when running Visual VM - since your heap was set to 512MB, you need to set the heap for VisualVM to around 1.5GB to 2GB, which means you might need to run with a 64-bit JVM.

                        • 9. Out of Memory in AS 5.1.0.GA
                          Jon Loe Newbie

                          Thanks for the followup.   I'm running with a 64 bit VM, so will run in VisualVM when I get a chance.  But, looking at the stack, it looked like it could be related to the display tag lib, which is on the page that hangs (not all pages hang with out of memory).   So, I stripped the display tab table (after looking at the src code in display tag) and it looks like if I change this it works:

                           

                          <display:column align='left' width='20%' titleKey='document.expiration.label'>

                           

                          to

                           

                          <display:column align='left' width='20%' title='Expiration Date'>

                           

                          Notice the title, titleKey element.   For some reason, display tag doesn't like the ref to the properties file.   so, going to research that some more.   will post my results when done...

                           

                          thanks