1 Reply Latest reply on Jul 3, 2014 2:15 AM by davsclaus

    OutOfMemoryError: Java heap space in profile with loadBalance

    divine_evil
    divine_evil Jul 2, 2014 6:21 AM

      Hi,

       

      We have JBoss FUSE 6.1

       

      I have very simple route

       

              <from uri="jetty:http...../service?matchOnUriPrefix=true" />

              <loadBalance inheritErrorHandler="false">

                  <failover roundRobin="true">

                      <exception>java.net.ConnectException</exception>

                      <exception>java.io.IOException</exception>

                      <exception>java.lang.Exception</exception>

                  </failover>

                  <to uri="http..../Service?bridgeEndpoint=true" />

                  <to uri="http..../Service?bridgeEndpoint=true" />

                  <to uri="http..../Service?bridgeEndpoint=true" />

                  <to uri="http..../Service?bridgeEndpoint=true" />

              </loadBalance>

       

      after 14 days container was stuck and log full of messages:

      java.lang.OutOfMemoryError: Java heap space

       

      2014-07-01 16:00:52,812 | WARN  | qtp503115302-254 | ServletHandler                   | pse.jetty.servlet.ServletHandler  590 | 185 - org.eclipse.jetty.aggregate.jetty-all-server - 8.1.14.v20131031 | Error for /myservice

      java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2271)[:1.7.0_51]

        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)[:1.7.0_51]

        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)[:1.7.0_51]

        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)[:1.7.0_51]

        at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:145)

        at org.apache.camel.util.IOHelper.copy(IOHelper.java:199)

        at org.apache.camel.util.IOHelper.copy(IOHelper.java:169)

        at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:218)

        at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:214)

        at org.apache.camel.component.http.helper.HttpHelper.readResponseBodyFromInputStream(HttpHelper.java:170)

        at org.apache.camel.component.http.helper.HttpHelper.readResponseBodyFromServletRequest(HttpHelper.java:149)

        at org.apache.camel.component.http.DefaultHttpBinding.parseBody(DefaultHttpBinding.java:447)

        at org.apache.camel.component.http.HttpMessage.createBody(HttpMessage.java:60)

        at org.apache.camel.impl.MessageSupport.getBody(MessageSupport.java:41)[110:org.apache.camel.camel-core:2.12.0.redhat-610379]

        at org.apache.camel.component.http.DefaultHttpBinding.readRequest(DefaultHttpBinding.java:82)

        at org.apache.camel.component.http.HttpMessage.<init>(HttpMessage.java:46)

        at org.apache.camel.component.jetty.CamelContinuationServlet.service(CamelContinuationServlet.java:115)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)

        at org.eclipse.jetty.servlets.MultiPartFilter.doFilter(MultiPartFilter.java:136)

        at org.apache.camel.component.jetty.CamelFilterWrapper.doFilter(CamelFilterWrapper.java:44)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:370)

        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)

        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)

      2014-07-01 16:25:24,293 | ERROR | cfailover/deploy | fileinstall                      | ?                                   ? | 7 - org.apache.felix.fileinstall - 3.3.11.redhat-610379 | In main loop, we have serious trouble

      java.lang.OutOfMemoryError: Java heap space

       

      most likely it was started by response from 1 of 4 servers:

      Error: Connection reset Stack trace:

        java.net.SocketException: Connection reset

       

       

      on all 4 target servers was lot of:

      Interceptor for {http:......./service}Service has thrown exception, unwinding now

      ·         org.apache.cxf.binding.soap.SoapFault: Error reading XMLStreamReader.
      ·         Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
       at [row,col {unknown-source}]: [1,0]

       

       

       

      my wrapper conf looks like that:

       

       

      wrapper.java.additional.13=-XX:PermSize=64M

      wrapper.java.additional.14=-XX:MaxPermSize=256M

       

      # Initial Java Heap Size (in MB)

      wrapper.java.initmemory=256

       

      # Maximum Java Heap Size (in MB)

      wrapper.java.maxmemory=1024

       

       

      I had to stop and start container to make it work again

       

      how can I prevent this situation ?

       

      It looks like after rejection of 1 server failover was trying to resend it again, but with some malformed request and it ends in endless loop.

       

       

       

      Thanks Radim

      • 1495 Views
      • Tags: