OutOfMemoryError: Java heap space in profile with loadBalance
divine_evil Jul 2, 2014 6:21 AMHi,
We have JBoss FUSE 6.1
I have very simple route
<from uri="jetty:http...../service?matchOnUriPrefix=true" />
<loadBalance inheritErrorHandler="false">
<failover roundRobin="true">
<exception>java.net.ConnectException</exception>
<exception>java.io.IOException</exception>
<exception>java.lang.Exception</exception>
</failover>
<to uri="http..../Service?bridgeEndpoint=true" />
<to uri="http..../Service?bridgeEndpoint=true" />
<to uri="http..../Service?bridgeEndpoint=true" />
<to uri="http..../Service?bridgeEndpoint=true" />
</loadBalance>
after 14 days container was stuck and log full of messages:
java.lang.OutOfMemoryError: Java heap space
2014-07-01 16:00:52,812 | WARN | qtp503115302-254 | ServletHandler | pse.jetty.servlet.ServletHandler 590 | 185 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.14.v20131031 | Error for /myservice
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)[:1.7.0_51]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)[:1.7.0_51]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)[:1.7.0_51]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)[:1.7.0_51]
at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:145)
at org.apache.camel.util.IOHelper.copy(IOHelper.java:199)
at org.apache.camel.util.IOHelper.copy(IOHelper.java:169)
at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:218)
at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:214)
at org.apache.camel.component.http.helper.HttpHelper.readResponseBodyFromInputStream(HttpHelper.java:170)
at org.apache.camel.component.http.helper.HttpHelper.readResponseBodyFromServletRequest(HttpHelper.java:149)
at org.apache.camel.component.http.DefaultHttpBinding.parseBody(DefaultHttpBinding.java:447)
at org.apache.camel.component.http.HttpMessage.createBody(HttpMessage.java:60)
at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:41)[110:org.apache.camel.camel-core:2.12.0.redhat-610379]
at org.apache.camel.component.http.DefaultHttpBinding.readRequest(DefaultHttpBinding.java:82)
at org.apache.camel.component.http.HttpMessage.<init>(HttpMessage.java:46)
at org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:115)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:136)
at org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:44)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
2014-07-01 16:25:24,293 | ERROR | cfailover/deploy | fileinstall | ? ? | 7 - org.apache.felix.fileinstall - 3.3.11.redhat-610379 | In main loop, we have serious trouble
java.lang.OutOfMemoryError: Java heap space
most likely it was started by response from 1 of 4 servers:
Error: Connection reset Stack trace:
java.net.SocketException: Connection reset
on all 4 target servers was lot of:
Interceptor for {http:......./service}Service has thrown exception, unwinding now
· org.apache.cxf.binding.soap.SoapFault: Error reading XMLStreamReader.
· Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
my wrapper conf looks like that:
wrapper.java.additional.13=-XX:PermSize=64M
wrapper.java.additional.14=-XX:MaxPermSize=256M
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=1024
I had to stop and start container to make it work again
how can I prevent this situation ?
It looks like after rejection of 1 server failover was trying to resend it again, but with some malformed request and it ends in endless loop.
Thanks Radim