2 Replies Latest reply on Nov 29, 2011 4:18 AM by grahamp

    Memory consumption

    grahamp

      We are using ESB to integrate systems that need intermittent file transfers.  File is usually small (0 - 5Kb) but occasionally large (2 - 4 Mb).  In the route this could transform into a 6Mb message which we then send to another system via a web API.

       

      Looking at the ubuntu system monitor, we see a single such file causing the ESB to go from 140Mb idle to 850Mb busy.  Or worse .. sometimes out-of-memory exception on a 1Gb JVM.  Why is camel routing consuming so much memory?  Only simple nodes in the route, no queues:

      - Processor to convert csv to xml

      - validate against an xsd

      - transform with an xslt

      - Processor to write to the OutputStream of a web-api post operation

      - Processor to check the response from the post operation

       

      We could understand a few 6Mb copies being generated during the exchange, but 700Mb overhead is a shock.  Would we gain anything by splitting the route up with file endpoints?

        • 1. Re: Memory consumption
          njiang

          Which version of Fuse ESB are you using?

          What kind of Web API are you using?

           

          We need more information to dig the issue.

          • 2. Re: Memory consumption
            grahamp

            We are using servicemix 4.3.1 and observe the behaviour in both versions fuse-03-01 (on ubuntu) and fuse-02-05 (windows server 2008).  Currently there is no web api in the bundle at all.  The routes simply pick up from network file areas.  They all start with something like

              from("file:////vmtest/esb_in")

             

            The increasing memory load happens with one file on one route - everything else idle.