1 Reply Latest reply on Oct 13, 2005 11:55 PM by Scott Stark

    Upgrade from 3->4 ; New OutOfMemoryError occuring

    Ian Hunter Newbie

      We're upgrading our application servers from Jboss3.2.4 to JBoss4.0.1. Currently, our web application runs fine (fine == 50k page requests and no crashes, GC working properly).

      However, after the upgrade to 4.0.1, the application still serves content but at about 25k page requests (25/second) it kills the server with an OutOfMemoryError.

      I would normally consider this to be a developer memory leak issue, but the codebase is IDENTICAL between the two server deployments. Running top on our redhat machines, I watch the mem usage rise to 1.6GB and then the problems start. It does not appear to be garbage collecting, which I guess it wouldn't if it still had active objects (or so it thinks).

      ===========================================

      The question is: Is something different about the XML/XSLT libaries between jboss3 and jboss4 that would either cause an inconsistency or introduce a memory leak in our deployment? If so, what would the appropriate jar/lib to update to?

      ===========================================

      System Knowledge:

      I threw JMemprof up on there and I'm not really getting a great amount of information. The peculiar piece of it is that the highest number of objects seem to be of type "unknown" (gee, thanks).

      Checking the method hierarchy, there are a LOT of xalan related method calls.

      The stack traces ALWAYS come from the xalan libraries. Here's an example:

      javax.xml.transform.TransformerException: java.lang.OutOfMemoryError
      at org.apache.xalan.extensions.ExtensionHandlerJavaPackage.callFunction(ExtensionHandlerJavaPackage.java:403)
      at org.apache.xalan.extensions.ExtensionHandlerJavaPackage.callFunction(ExtensionHandlerJavaPackage.java:426)
      at org.apache.xalan.extensions.ExtensionsTable.extFunction(ExtensionsTable.java:220)
      at org.apache.xalan.transformer.TransformerImpl.extFunction(TransformerImpl.java:437)
      at org.apache.xpath.functions.FuncExtFunction.execute(FuncExtFunction.java:199)
      at org.apache.xpath.XPath.execute(XPath.java:268)
      at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:279)
      at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemElement.constructNode(ElemElement.java:338)
      at org.apache.xalan.templates.ElemElement.execute(ElemElement.java:287)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemIf.execute(ElemIf.java:161)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
      at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
      at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
      at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
      at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
      at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(TransformerImpl.java:2202)
      at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1276)
      at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:673)
      at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1192)
      at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1170)
      at com.mycompany.SpecificResponse.applyXslt(Unknown Source)
      at com.mycompany.SpecificServlet.service(Unknown Source)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:66)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:150)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:54)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
      at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
      at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.lang.OutOfMemoryError

      Obviously I changed the company name and class names.

      I've tried a number of XSLT transformation optimizations:

      1) static the TransformerFactory instance

      2) hashmap cache the Templates objects so they can be used again, if possible

      3) tFactory.setAttribute
      ("http://xml.apache.org/xalan/features/incremental",
      java.lang.Boolean.TRUE);
      as was suggested by:
      http://xml.apache.org/xalan-j/dtm.html#incremental

      Most of these were already in our codebase before the switch.

      ===========================================

      Any help would be most appreciated.

      ===========================================

      -ian