We're upgrading our application servers from Jboss3.2.4 to JBoss4.0.1. Currently, our web application runs fine (fine == 50k page requests and no crashes, GC working properly).

However, after the upgrade to 4.0.1, the application still serves content but at about 25k page requests (25/second) it kills the server with an OutOfMemoryError.

I would normally consider this to be a developer memory leak issue, but the codebase is IDENTICAL between the two server deployments. Running top on our redhat machines, I watch the mem usage rise to 1.6GB and then the problems start. It does not appear to be garbage collecting, which I guess it wouldn't if it still had active objects (or so it thinks).

===========================================

The question is: Is something different about the XML/XSLT libaries between jboss3 and jboss4 that would either cause an inconsistency or introduce a memory leak in our deployment? If so, what would the appropriate jar/lib to update to?

===========================================

System Knowledge:

I threw JMemprof up on there and I'm not really getting a great amount of information. The peculiar piece of it is that the highest number of objects seem to be of type "unknown" (gee, thanks).

Checking the method hierarchy, there are a LOT of xalan related method calls.

The stack traces ALWAYS come from the xalan libraries. Here's an example:

javax.xml.transform.TransformerException: java.lang.OutOfMemoryError
at org.apache.xalan.extensions.ExtensionHandlerJavaPackage.callFunction(ExtensionHandlerJavaPackage.java:403)
at org.apache.xalan.extensions.ExtensionHandlerJavaPackage.callFunction(ExtensionHandlerJavaPackage.java:426)
at org.apache.xalan.extensions.ExtensionsTable.extFunction(ExtensionsTable.java:220)
at org.apache.xalan.transformer.TransformerImpl.extFunction(TransformerImpl.java:437)
at org.apache.xpath.functions.FuncExtFunction.execute(FuncExtFunction.java:199)
at org.apache.xpath.XPath.execute(XPath.java:268)
at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:279)
at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemElement.constructNode(ElemElement.java:338)
at org.apache.xalan.templates.ElemElement.execute(ElemElement.java:287)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:127)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemIf.execute(ElemIf.java:161)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:682)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.templates.ElemTemplate.execute(ElemTemplate.java:393)
at org.apache.xalan.templates.ElemCallTemplate.execute(ElemCallTemplate.java:247)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336)
at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(TransformerImpl.java:2202)
at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1276)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:673)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1192)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1170)
at com.mycompany.SpecificResponse.applyXslt(Unknown Source)
at com.mycompany.SpecificServlet.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:66)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:150)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:54)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.lang.OutOfMemoryError

Obviously I changed the company name and class names.

I've tried a number of XSLT transformation optimizations:

1) static the TransformerFactory instance

2) hashmap cache the Templates objects so they can be used again, if possible

3) tFactory.setAttribute
("http://xml.apache.org/xalan/features/incremental",
java.lang.Boolean.TRUE);
as was suggested by:
http://xml.apache.org/xalan-j/dtm.html#incremental

Most of these were already in our codebase before the switch.

===========================================

Any help would be most appreciated.

===========================================

-ian