Problem using dynamic fileName with Camel File Component

[annasw]

Is it possible to use camel file component for polling files based on fileName parameter in URI?

Like: <from uri="file://data?&amp;fileName=$"/>

I am trying to get camel file component polling the file based on fileName=$ parameter: <from uri="file://data?noop=true&amp;fileName=$"/>

For some reason it never camel polls the file. Should it work with dynamic parameter?

(It only pools if I use hard coded file name inside of my routes:

<from uri="file://data?noop=true&amp;fileName=$"/>

public void process(Exchange exchange) throws Exception

{

   exchange.getOut().setHeader(Exchange.FILE_NAME, myDynamicFileName);          

}

Thank you very much for your time,

Anna

Edited by: asilchenko on May 25, 2010 1:29 PM

Edited by: asilchenko on May 25, 2010 1:30 PM

Edited by: asilchenko on May 25, 2010 1:42 PM

Edited by: asilchenko on May 25, 2010 1:53 PM

Edited by: asilchenko on May 25, 2010 4:34 PM

[annasw]

After trying a few things got it working using system properties ${sys.fileName} instead of $

But still have issue with the slowness. Is there a way to make Camel File Component return file faster?

Thank you very much!

Edited by: asilchenko on May 25, 2010 11:40 PM

Edited by: asilchenko on May 25, 2010 11:42 PM

[davsclaus]

Hi

You can use the delay option to control how fast the file component should check for new files.

http://camel.apache.org/file2.html

[annasw]

Hi Claus,

Thank you for your response. I am still having slowness in Camel File Component to return files identified by fileName=${sys.fileName}. Here is my route:

Could you please let me know if there is anything else I can try to get it work faster? (I am calling this route about 30-50 times per second. So far, from what I have seeing, Camel File Component returns file at about 10 files per second rate.) Thank you very much for your time and help!

Anna

Edited by: asilchenko on Jun 1, 2010 1:08 PM

Edited by: asilchenko on Jun 1, 2010 1:09 PM

[davsclaus]

You send the File to an AMQ queue which takes time.

So remove that part and send it to a log instead which then should run fast.

<to uri="log:foo"/>

And if that runs fast, then you know its the send the file to the AMQ which take a bit of time. The file consumer is running in a single thread so it takes one file at a time and send that to the AMQ, and when that is done. It pickup the next file.

You can find a blog entry on my blog how to speed up this using concurrency.

http://davsclaus.blogspot.com/

I dont have time to dig it out, but it was a while back, maybe 6+ months or older.

[annasw]

Thank you for your response. I tried a couple of things:

1)  Sent messages to

I found blog about async and threads:

http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html

I would like to try threads or async with Camel File Component but could not find syntax for  Spring DSL.

Could you please point me or let me know Spring DSL syntax for threads or async? Thank you so much!

Anna

[davsclaus]

Dont you have an Java editor which can assist you when doing XML editing?

There is a XSD which lists the EIPs you can use, eg in Spring it should be

<from .../>
<threads>
  <to uri=...></to>
</threads>

[annasw]

Thank you so much for your response and the snippet. Unfortunately, getting same slowness issue even with .

<from uri="file://data?noop=true&amp;delay=1&amp;fileName=${sys.fileName}"/>
<threads>
<to uri="activemq:topic:DATA_OUTPUT_TEST"></to>
</threads>

But if use regular java code instead of Camel File Component getting no slowness.

<bean ref="fileReader" method="readFileContentForGivenFileName"/>
<to uri="activemq:topic:DATA_OUTPUT_TEST"></to>

Do you think it is possible for this to be an issue with Camel File Component?

Does Camel File Component have any preferences or restrictions regarding file names format? My files named like this: "20100526T202946.000134Z"

Thank you very much,

Anna

Edited by: asilchenko on Jun 2, 2010 5:23 PM

Edited by: asilchenko on Jun 2, 2010 5:28 PM

Edited by: asilchenko on Jun 2, 2010 5:29 PM

Edited by: asilchenko on Jun 2, 2010 5:34 PM

[davsclaus]

Can you create a small project as a ZIP file then I can take a look at it to see if there is any mistakes in the camel file consumer.

[annasw]

Hi Claus,

Thank you so much in advance for looking into this! I created a simple maven project which sort of emulates the usage of the Camel File Component (CFC) in my "real" project and I was able to reproduce the slowness. Please see my attached project in playback.zip and feel free to build and run it using following commands:

mvn clean -Pone-jar clean package
java -jar target/playback-0.0.1-SNAPSHOT.one-jar.jar

My routes located in resources/routes.xml file. I also have routes2.xml for reference which use plain Java instead of CFC. When using routes2.xml there is no slowness.

Also, in the attached log file, you may see information how many files/messages should be send and how many were received. In the example below send is 141 and receive is 57.

INFO - Camel File component should send file count/name  = 141/20100526T203051.000136Z
INFO - MyReceiver received message count/content 57/20100526T201551.000125Z

So only 57 out of 141 were received. But if I use routes2.xml, there no messages lost or delayed.

Thank you so much for all your time and help!

Best Regards,

Anna

Edited by: asilchenko on Jun 7, 2010 9:07 PM

Edited by: asilchenko on Jun 10, 2010 2:08 PM

[davsclaus]

I am bit to busy at the moment, but got this issue logged and will take a look when time clears in the near future.

[davsclaus]

I did profile a bit yesterday and I could not find any obvious bottleneck.

Of course doing an optimize hand code will be faster as you do not do all the extra work Camel does.

Do you have a large number of files in the directory to pickup? And what OS are you using?

[annasw]

I have 140 files in the directory. I am running CentOS 4.7.

Thanks,

Anna

[davsclaus]

I created an unit test which is fast to poll 200 files (from one directory)

2010-07-14 11:23:10,117 DEBUG FileConsumer                   - Took 0.010 seconds to pool: target/manyfiles

Your 140 files are they located in the same directory? Or in many different directorties?

I have added the timing to the source code which then will be in FUSE MR 2.4.0 or Camel 2.5.0.