A design question

[simplex-software]

Greetings,

We have the following very specific situation in a data-loading process. We receive a text file containing a list of files to be processed. these files are supposed to be already available on the file-system but we don't know where before receiving the text file.

The first step would be to split the file, line by line, as each line contain the path and name of one file. Then the file which name is in the message body has to be processed, so I need to dynamiclly define a route to pick-up this file.

And it wouldn't be really a route as it is not repetable, the file having the given path and name being processed only once and no other file will have the same path and name.

I know that these requirements don't seem too mature, but well, we need to deal with that. The point is that I don't see how to dynamically define route which endpoints are not known and which, besides that, are even not routes, but only one shot processes.

Many thanks in advance for any idea.

Kind regards,

Nicolas

[davsclaus]

You can use the content enricher EIP to enrich the splitted message with the file

http://camel.apache.org/content-enricher.html

But the issue is that this EIP currently does not support specifying a dynamic URI with information from the current exchange. This will be improved in the future.

So what you can do instead, is just to set the message body to the file handle of the file to pickup, for example from a method call, and then use the message translator EIP and invoke the method.

public File pickupFile(String name) {
  return new File(name);
}

Then the only question remains, should you do something with the file after you have successfully processed it? eg should you move or delete the file, or nothing at all? If you have to do something, then you would need to do this manually, for example using java code, to eg delete the file etc.

[simplex-software]

Hi Claus,

Many thanks for your answer. I'm not sure I understood what you meant but I came at the following code:

    from("file:input").split().tokenize("\n").process(new FileConsumer()).bean(FileProcessor.class);

public class FileConsumer implements Processor

{

  private static Logger LOG = Logger.getLogger(FileConsumer.class);

  @Override

  public void process(Exchange exchg) throws Exception

  {

    ConsumerTemplate ct = exchg.getContext().createConsumerTemplate();

    String filePath = (String)exchg.getIn().getBody();

    Endpoint ep = exchg.getContext().getEndpoint("file:input/files?fileName=" + filePath.substring(0, filePath.length()-2));

    exchg = ct.receive(ep);

  }

}

The meta-file, ie the file containing file names, will be picked-up and splitted line by line. each line will be passed, as a string, to the FileConsumer which instantiates an endpoint and on its behalf consumes the file and passes it to the next node in the route.

What do you think, is that a good design ?

Kind regards,

Nicolas

[davsclaus]

Hi

I think your design is a bit overkill, and its also being dependent on the Camel API.

For example you could have a java bean with this code

public class FileConsumer {

public String loadFile(String filePath) {
String path = filePath.substring(0, filePath.length()-2);
return new File("input/files/" + path));
}

And then use that in the route

from("file:input")
  .split().tokenize("\n")
    .bean(FileConsumer.class)
    .bean(FileProcessor.class);

[davsclaus]

But it also depends a bit what you do in the FileProcessor? As with my code you essentially set the message body to a java.io.File handle. Where as with your code, you use the ConsumerTemplate API to poll a message (which happens to be file based).

The latter supports the various options on the file component of Camel, so you automatic can move/delete the file afterwards, where as in the first solution you will have to do this yourself.

Also ideally you should not create a new consumer template all the time. See this FAQ as it also applies for the consumer template

http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html

[simplex-software]

Hi Claus,

Yes, I understand now.

Many thanks.

Nicolas