Exec endpoint is slow, how to use exec without blocking?

[nhavis]

Hi all,

I have an exec endpoint that calls python to run a script - the problem is that I have thousands of files coming in and the python script is supposed to run on each file.

The operation is far slower than normal and I suspect it's because exec is blocking and waiting for each script to finish before executing the next one. How can I get around this? I didn't find any endpoint options.

Thanks

[jorgemoralespou_2]

Hi Anthony,

The exec component is a camel component, so you should look into camel documentation for a solution to your problem. I guess you have added it as a SwitchYard extension, so you can use the component, so you should have some knowledge on how to use camel to do Synchronous Processing.

As of now, the exec component (http://camel.apache.org/exec.html) is synchronous, so yes, it is waiting to finish the execution of the scripts. If you need to do Asynchronous processing, split, fork/join or any other EIP to execute concurrently your scripts, you should look into the appropriate EIPs (http://camel.apache.org/enterprise-integration-patterns.html) provided by camel. (http://camel.apache.org/asynchronous-processing.html)

Cheers,

[nhavis]

Thanks, but are you able to give me more specifics on the appropriate EIP?

Literally all I'm doing is taking in files e.g. file1.ext and running script.py file1.ext