-
1. Out of Memory in AS 5.1.0.GA
peterj Jan 20, 2011 4:12 PM (in response to jnl1)JBoss AS 5.1.0 is know to use more memory than 4.2.3. I recommend increasing your heap size.
Other than that, you would have to look at some heap dumps to see what is using up the space.
By the way, there are several different reasons you might get a OOME. Since you didn't post the exact stack trace, I made a guess as to which OOME you got. But I could be wrong in which case my advice is useless.
-
2. Out of Memory in AS 5.1.0.GA
jnl1 Jan 20, 2011 4:13 PM (in response to jnl1)I took out the crypto information and still get the outOfMemory exception. So, not sure why 5.1 is so different thatn 4.2.3 ??....
-
3. Out of Memory in AS 5.1.0.GA
jnl1 Jan 20, 2011 4:16 PM (in response to jnl1)thanks for the quick response. sorry, should have included the stack trace:
11:25:58,731 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space
11:25:58,731 WARN [HDScanner] Scan failed
java.lang.OutOfMemoryError: Java heap space
11:26:01,968 ERROR [STDERR] at java.util.WeakHashMap.put(WeakHashMap.java:416)
11:26:01,969 ERROR [STDERR] at org.jboss.aop.AspectManager.getAdvisors(AspectManager.java:546)
11:26:04,884 ERROR [STDERR] at org.jboss.console.plugins.AOPLister$RefreshPoller.run(AOPLister.java:891)
11:26:04,885 ERROR [STDERR] Exception in thread "RMI TCP Connection(idle)"
-
4. Out of Memory in AS 5.1.0.GA
peterj Jan 20, 2011 4:20 PM (in response to jnl1)Hey, I guessed right!
-
5. Out of Memory in AS 5.1.0.GA
jnl1 Jan 20, 2011 4:51 PM (in response to peterj)Getting the same error with these params:
-Xms1024m -Xmx1024m -XX:MaxPermSize=512m
think I should try bumping it up more ?
-
6. Out of Memory in AS 5.1.0.GA
peterj Jan 20, 2011 5:06 PM (in response to jnl1)No. You have a memory leak and you have to find out what is leaking. Try adding -XX:+HeapDumpOnOutOfMemoryError to the JAVA_OPTS. This will cause the JVM to dump the heap when it gets an OOME. You can load the dump file into VisualVM (or some similar tool) and see what it taking up the space.
-
7. Out of Memory in AS 5.1.0.GA
jnl1 Jan 20, 2011 8:56 PM (in response to peterj)here's the stack trace.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to java_pid9460.hprof ...
Heap dump file created [517770450 bytes in 5.585 secs]
17:16:47,120 WARN [HDScanner] Scan failed
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:823)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:191)
at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
ava:684)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
ava:689)
at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
at org.jboss.deployers.vfs.spi.structure.modified.UpdateDeleteVisitor.doVisit(UpdateDeleteVisitor.java:53)
at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
or.java:62)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
at org.jboss.virtual.VFS.visit(VFS.java:421)
at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
rapperModificationChecker.java:90)
at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
AbstractStructureModificationChecker.java:195)
17:16:47,120 ERROR [STDERR] Exception in thread "ZipFile Lock Reaper"
17:16:47,132 ERROR [STDERR] java.lang.OutOfMemoryError: GC overhead limit exceeded
17:16:47,132 ERROR [STDERR] at java.util.concurrent.ConcurrentLinkedQueue.iterator(ConcurrentLinkedQueue.java:452)
17:16:47,133 ERROR [STDERR] at org.jboss.virtual.plugins.context.zip.ZipFileLockReaper$ReaperTimerTask.run(ZipFileLo
ckReaper.java:158)
17:16:47,133 ERROR [STDERR] at java.util.TimerThread.mainLoop(Timer.java:512)
17:16:47,133 ERROR [STDERR] at java.util.TimerThread.run(Timer.java:462)
17:17:02,741 WARN [HDScanner] Scan failed
java.lang.OutOfMemoryError: Java heap space
at java.lang.String.substring(String.java:1939)
at org.jboss.virtual.plugins.vfs.helpers.PathTokenizer.getTokens(PathTokenizer.java:135)
at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
ava:650)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
ava:689)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.j
ava:689)
at org.jboss.virtual.plugins.context.file.FileHandler.getChild(FileHandler.java:303)
at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:481)
at org.jboss.deployers.vfs.spi.structure.modified.AddVisitor.doVisit(AddVisitor.java:53)
at org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(SynchVisitor.java:66)
at org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(WrappingVirtualFileHandlerVisit
or.java:62)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:361)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:374)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:376)
at org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:306)
at org.jboss.virtual.VFS.visit(VFS.java:421)
at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
at org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(SynchW
rapperModificationChecker.java:93)
at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
AbstractStructureModificationChecker.java:195)
at org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(
AbstractStructureModificationChecker.java:138)
at org.jboss.system.server.profileservice.repository.HotDeploymentRepository.getModifiedDeployments(HotDeploymen
tRepository.java:120)
at org.jboss.system.server.profile.repository.AbstractProfile.getModifiedDeployments(AbstractProfile.java:128)
17:17:08,171 WARN [ClassLoaderManager] Unexpected error during load of:javax.servlet.jsp.jstl.fmt.LocalizationContext
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
17:17:12,571 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:261)
at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getRealURL(ZipEntryHandler.java:258)
at org.jboss.virtual.plugins.context.DelegatingHandler.getRealURL(DelegatingHandler.java:222)
at org.jboss.virtual.VFSUtils.getRealURL(VFSUtils.java:1064)
at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:651)at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:547)
at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507)
at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251)
at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.displaytag.localization.I18nJstlAdapter.getResource(I18nJstlAdapter.java:143)
at org.displaytag.tags.ColumnTag.addHeaderToTable(ColumnTag.java:609)
at org.displaytag.tags.ColumnTag.doEndTag(ColumnTag.java:547)
at org.apache.jsp.protected_.study.studyList_jsp._jspx_meth_display_005fcolumn_005f0(studyList_jsp.java:1851)
at org.apache.jsp.protected_.study.studyList_jsp._jspService(studyList_jsp.java:502)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
17:17:16,686 WARN [HDScanner] Scan failed
java.lang.OutOfMemoryError: Java heap space
17:17:20,986 ERROR [[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: Java heap space
17:17:20,987 ERROR [[localhost]] Exception Processing ErrorPage[exceptionType=java.lang.Throwable, location=/protected/e
rror.jsp]
javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:257)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:446)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:416)
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:270)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap spacethanks
-
8. Out of Memory in AS 5.1.0.GA
peterj Jan 21, 2011 11:25 AM (in response to jnl1)Ahah! The plot thickens! A "GC overhead limit exceeded" error happens because the GC is not cleaning up enough objects from the heap. There are two possible causes I can think of. One is that there is a memory leak. Actually, more than a memory leak - for some resons the code is hanging on to every object it is creating and never releasing anything. The other is that the GC settings are such that the GC finds it impossible to do its job.
The second is easier to fix, if that is the case. So what are your GC settings? Posting the full set of JVM options would help (they are displayed right when you first start up the app server).
As far as the memory leak goes, have you loaded java_pid9460.hprof into VisualVM? What did you see? If you haven't, be warned that you need about 3 to 4 times as much heap space when running Visual VM - since your heap was set to 512MB, you need to set the heap for VisualVM to around 1.5GB to 2GB, which means you might need to run with a 64-bit JVM.
-
9. Out of Memory in AS 5.1.0.GA
jnl1 Jan 22, 2011 9:36 PM (in response to peterj)Thanks for the followup. I'm running with a 64 bit VM, so will run in VisualVM when I get a chance. But, looking at the stack, it looked like it could be related to the display tag lib, which is on the page that hangs (not all pages hang with out of memory). So, I stripped the display tab table (after looking at the src code in display tag) and it looks like if I change this it works:
<display:column align='left' width='20%' titleKey='document.expiration.label'>
to
<display:column align='left' width='20%' title='Expiration Date'>
Notice the title, titleKey element. For some reason, display tag doesn't like the ref to the properties file. so, going to research that some more. will post my results when done...
thanks