Out Of Memory error on using xpath for content based router

[gedda_mallik]

Hi All,

We are trying to route documents generated from multiple external sources using content based router.

Everything works fine as long as the documents are small, but once the documents reach the size of 100mb

or so, the component throws out of memory exception(we will use much larger documents in prod).

Is there a way to address this or an alternative approach I can use to route documents based on content?

Much Thanks.

[davsclaus]

Can you limit the number of concurrent messages you route? To keep memory usages lower?

Or can you increase your memory settings on the JVM?

Or if your xpath expression is a fairly simple then you can try to use some SAX based parsers to grab the needed pieces. SAX based do not use much memory as they dont build up a DOM model.

And I think todays something as StAX is even faster.

And I wonder if the XPath stuff in the JDK can be tweaked / configured to use some less memory intensive?

[gedda_mallik]

The current flow of our process is

FilePoller --> eip(content based router) --> bean

An external system generates a large xml document which is polled and picked

up by the File Poller and hands it over to content based router(which runs some

xpath queries) and hands it over to the correct end point based on the routing

rules.

We can control the process to use sax parser at the bean end point

but unfortunately the process fails even before reaching the bean end point.

To rule out which component is causing the issue we have removed the eip

component and now the process is

FilePoller --> bean

Even now the process fails with out of memory exception even before

reaching the bean end point.

We have increased the memory to the jvm (1536M)

So we are not sure if the process is failing trying to

build the message exchange or due to some other reason

Any help is greatly appreciated.

[njiang]

Can I see your bean's method?

If your just has the parameter with an InputStream, you will get into trouble of Out  of Memory. It is because camel will try to create a BufferedInputStream from the InMessage body for your beam method to use.

You can choice the Exchange , or the File as the bean method's parameter, they will use much less memory

[gedda_mallik]

As mentioned earlier the process is pretty basic

filepoller --> eip -->bean

Please find the attached bean java file.

The exception is happening even before the onMessageExchange method is called

on this attached file

[davsclaus]

Ah you are using Fuse ESB (ServiceMix).

It would be good if you can post the versions you are using and which JDK and OS you are using?

This forum is for Camel (Mediation Router) so we thought you was using Camel.

[gedda_mallik]

I'm so new to this, thought that the exception is happening at the servicemix-eip

component, so I have posted in this forum. Please move it to the correct forum if

required.

We are using jdk 1.6 , Red Hat 4.7 and running on 1536M of memory

and fuse-esb-3.4.0.4

[gedda_mallik]

Do I need to post the same question in Fuse ESB forum?

or Can this post be moved to the other forum?

Any help is greatly appreciated.

[davsclaus]

You could post a link on the other forum with a link to this thread to get the attention of the FUSE ESB people.

[gedda_mallik]

Thank you