13 Replies Latest reply on Mar 7, 2013 5:02 AM by mari_yaguchi

Camel Route Transaction Per Second so low

mari_yaguchi Mar 4, 2013 11:27 PM

Dear all, i just learn about Camel to route some files in Fuse ESB. But the result I get from my test is the performance is very low. The TPS i got is very low, i got just about 15-20 tps for routing file from and to the same local disk. Can you help me how to higher up this number please? can you share me about your tps you got with camel route? Thanks for your help.

1. Re: Camel Route Transaction Per Second so low

davsclaus Mar 5, 2013 4:24 AM (in response to mari_yaguchi)

What version of Fuse ESB do you use.

I assume you use Camel to route the files. If so what is your Camel route?
And how big is the files being routed? Are they being routed to the same disk or to another SAN etc.

And if you move/copy the files using the file system manually does that run faster?

And what file system do you use?
Actions
2. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 5, 2013 4:47 AM (in response to davsclaus)

Thanks for your reply davsclaus, really appreciate it.

I use FuseESB 7.0.2 and I use camel route to route the files. My camel route configuration is :

The file size is just 1kb and they are routed to the same disk. If I use the file system to copy the files manually, it is faster. I use Centos 6 32bit with 8GB RAM Intel Xeon 3,1GHz as the environment to run Fuse.
Actions
3. Re: Camel Route Transaction Per Second so low

davsclaus Mar 5, 2013 11:47 AM (in response to mari_yaguchi)

How many files do you have in the Source236 directory?
Actions
4. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 5, 2013 8:37 PM (in response to davsclaus)

There are 500 files in Source236 directory..
Actions
5. Re: Camel Route Transaction Per Second so low

davsclaus Mar 6, 2013 3:04 AM (in response to mari_yaguchi)

Can you try removing

to
Actions
6. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 6, 2013 3:46 AM (in response to davsclaus)

Thanks Claus,
I used the log message to measure the TPS, so i can't measure the TPS without log message. Do you have any suggestion to measure the tps without using log message?

Edit :
I tried your suggestion to remove the log message, but it still take the same time to route all file like routing with the log message. No change.

Edited by: mari_yaguchi on Mar 6, 2013 8:44 AM
Actions
7. Re: Camel Route Transaction Per Second so low

davsclaus Mar 6, 2013 8:59 AM (in response to mari_yaguchi)

And if you write some Java code to move the files. Can you write this code so it runs faster?

Mind that Camel moves a file one by one.

Where as if you move the files from the OS shell it may move in bulks / being optimized, if you do a move with wildcard.

Also you can enable TRACE/DEBUG logging on org.apache.camel.component.file, which logs some information what is going on. Maybe something is revealed there.
Actions
8. Re: Camel Route Transaction Per Second so low

davsclaus Mar 6, 2013 9:00 AM (in response to mari_yaguchi)

And I assume those 500 files is already in the folder when you start the Camel route?

The file consumer in Camel will by default run with 500 milli seconds interval after its last run.
Actions
9. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 6, 2013 9:13 AM (in response to davsclaus)

Thanks for Claus for the answer, I dont use any java code here, because I use the hot deployment. So i just put the xml camel configuration to the deploy folder.
Those 500 files is already in the source folder.
Is there any way to change the default 500ms interval?
Actions
10. Re: Camel Route Transaction Per Second so low

davsclaus Mar 6, 2013 9:53 AM (in response to mari_yaguchi)

Yeah its the delay option
http://camel.apache.org/file2
Actions
11. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 6, 2013 8:40 PM (in response to davsclaus)

If the default value is 500ms, so I just can get 2 files being routed in 1 second?is that right? if i change it to 100ms then I will get 10 files routed in 1 second? so the TPS is 10? is it right?
Actions
12. Re: Camel Route Transaction Per Second so low

davsclaus Mar 7, 2013 4:06 AM (in response to mari_yaguchi)

No the delay is the time between polls where Camel will check for new files arriving. So if there is 27 new files then all 27 files is processed. And then Camel sleep for the delay duration, and then check for new files again.

You can enable DEBUG or TRACE logging on org.apache.camel.component.file to see more details. Maybe there is something that shows why its "slow" on your system.

Also mind that when using noop=true then Camel the files is left as is, so when Camel runs again, it has to check if the file has been processed before or not. This can get slower if you have a lot of files in the directory, eg 10000+ or more files. But if you only have 500 files then that ought to be fast.

Anyway if you write some Java code and run on that box to compare if you can write some code that is much faster, then that's interesting. Camel can only be as fast as the Java / JDK allows it to do.
Actions
13. Re: Camel Route Transaction Per Second so low

mari_yaguchi Mar 7, 2013 5:02 AM (in response to davsclaus)

Thanks Claus for your advice, really appreciate it. Anyway, is it a good thing if I use Seda here? So i route the files to seda first, then i route the files in seda to another folder, is it applicable?
Actions

Go to original post