-
1. Re: Problem using dynamic fileName with Camel File Component
annasw May 25, 2010 7:42 PM (in response to annasw)After trying a few things got it working using system properties ${sys.fileName} instead of $
But still have issue with the slowness. Is there a way to make Camel File Component return file faster?
Thank you very much!
Edited by: asilchenko on May 25, 2010 11:40 PM
Edited by: asilchenko on May 25, 2010 11:42 PM
-
2. Re: Problem using dynamic fileName with Camel File Component
davsclaus May 26, 2010 1:41 AM (in response to annasw)Hi
You can use the delay option to control how fast the file component should check for new files.
-
3. Re: Problem using dynamic fileName with Camel File Component
annasw Jun 1, 2010 9:09 AM (in response to davsclaus)Hi Claus,
Thank you for your response. I am still having slowness in Camel File Component to return files identified by fileName=${sys.fileName}. Here is my route:
Could you please let me know if there is anything else I can try to get it work faster? (I am calling this route about 30-50 times per second. So far, from what I have seeing, Camel File Component returns file at about 10 files per second rate.) Thank you very much for your time and help!
Anna
Edited by: asilchenko on Jun 1, 2010 1:08 PM
Edited by: asilchenko on Jun 1, 2010 1:09 PM
-
4. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jun 1, 2010 10:16 AM (in response to annasw)You send the File to an AMQ queue which takes time.
So remove that part and send it to a log instead which then should run fast.
<to uri="log:foo"/>
And if that runs fast, then you know its the send the file to the AMQ which take a bit of time. The file consumer is running in a single thread so it takes one file at a time and send that to the AMQ, and when that is done. It pickup the next file.
You can find a blog entry on my blog how to speed up this using concurrency.
http://davsclaus.blogspot.com/
I dont have time to dig it out, but it was a while back, maybe 6+ months or older.
-
5. Re: Problem using dynamic fileName with Camel File Component
annasw Jun 1, 2010 3:00 PM (in response to davsclaus)Thank you for your response. I tried a couple of things:
1) Sent messages to
I found blog about async and threads:
http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html
I would like to try threads or async with Camel File Component but could not find syntax for Spring DSL.
Could you please point me or let me know Spring DSL syntax for threads or async? Thank you so much!
Anna
-
6. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jun 2, 2010 1:17 AM (in response to annasw)Dont you have an Java editor which can assist you when doing XML editing?
There is a XSD which lists the EIPs you can use, eg in Spring it should be
<from .../> <threads> <to uri=...></to> </threads>
-
7. Re: Problem using dynamic fileName with Camel File Component
annasw Jun 2, 2010 1:36 PM (in response to davsclaus)Thank you so much for your response and the snippet. Unfortunately, getting same slowness issue even with .
<from uri="file://data?noop=true&delay=1&fileName=${sys.fileName}"/> <threads> <to uri="activemq:topic:DATA_OUTPUT_TEST"></to> </threads>
But if use regular java code instead of Camel File Component getting no slowness.
<bean ref="fileReader" method="readFileContentForGivenFileName"/> <to uri="activemq:topic:DATA_OUTPUT_TEST"></to>
Do you think it is possible for this to be an issue with Camel File Component?
Does Camel File Component have any preferences or restrictions regarding file names format? My files named like this: "20100526T202946.000134Z"
Thank you very much,
Anna
Edited by: asilchenko on Jun 2, 2010 5:23 PM
Edited by: asilchenko on Jun 2, 2010 5:28 PM
Edited by: asilchenko on Jun 2, 2010 5:29 PM
Edited by: asilchenko on Jun 2, 2010 5:34 PM
-
8. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jun 3, 2010 1:57 AM (in response to annasw)Can you create a small project as a ZIP file then I can take a look at it to see if there is any mistakes in the camel file consumer.
-
9. Re: Problem using dynamic fileName with Camel File Component
annasw Jun 10, 2010 10:10 AM (in response to davsclaus)Hi Claus,
Thank you so much in advance for looking into this! I created a simple maven project which sort of emulates the usage of the Camel File Component (CFC) in my "real" project and I was able to reproduce the slowness. Please see my attached project in playback.zip and feel free to build and run it using following commands:
mvn clean -Pone-jar clean package java -jar target/playback-0.0.1-SNAPSHOT.one-jar.jar
My routes located in resources/routes.xml file. I also have routes2.xml for reference which use plain Java instead of CFC. When using routes2.xml there is no slowness.
Also, in the attached log file, you may see information how many files/messages should be send and how many were received. In the example below send is 141 and receive is 57.
INFO - Camel File component should send file count/name = 141/20100526T203051.000136Z INFO - MyReceiver received message count/content 57/20100526T201551.000125Z
So only 57 out of 141 were received. But if I use routes2.xml, there no messages lost or delayed.
Thank you so much for all your time and help!
Best Regards,
Anna
Edited by: asilchenko on Jun 7, 2010 9:07 PM
Edited by: asilchenko on Jun 10, 2010 2:08 PM
-
playback.log 98.6 KB
-
playback.zip 39.8 KB
-
-
10. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jun 10, 2010 10:38 AM (in response to annasw)I am bit to busy at the moment, but got this issue logged and will take a look when time clears in the near future.
-
11. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jun 14, 2010 1:04 AM (in response to davsclaus)I did profile a bit yesterday and I could not find any obvious bottleneck.
Of course doing an optimize hand code will be faster as you do not do all the extra work Camel does.
Do you have a large number of files in the directory to pickup? And what OS are you using?
-
12. Re: Problem using dynamic fileName with Camel File Component
annasw Jun 14, 2010 10:14 AM (in response to davsclaus)I have 140 files in the directory. I am running CentOS 4.7.
Thanks,
Anna
-
13. Re: Problem using dynamic fileName with Camel File Component
davsclaus Jul 14, 2010 6:02 AM (in response to annasw)I created an unit test which is fast to poll 200 files (from one directory)
2010-07-14 11:23:10,117 DEBUG FileConsumer - Took 0.010 seconds to pool: target/manyfiles
Your 140 files are they located in the same directory? Or in many different directorties?
I have added the timing to the source code which then will be in FUSE MR 2.4.0 or Camel 2.5.0.