5 Replies Latest reply on Jun 22, 2012 9:06 AM by nicolas duminil

    A design question

    nicolas duminil Master

      Greetings,

       

      We have the following very specific situation in a data-loading process. We receive a text file containing a list of files to be processed. these files are supposed to be already available on the file-system but we don't know where before receiving the text file.

       

      The first step would be to split the file, line by line, as each line contain the path and name of one file. Then the file which name is in the message body has to be processed, so I need to dynamiclly define a route to pick-up this file.

       

      And it wouldn't be really a route as it is not repetable, the file having the given path and name being processed only once and no other file will have the same path and name.

       

      I know that these requirements don't seem too mature, but well, we need to deal with that. The point is that I don't see how to dynamically define route which endpoints are not known and which, besides that, are even not routes, but only one shot processes.

       

      Many thanks in advance for any idea.

       

      Kind regards,

       

      Nicolas

        • 1. Re: A design question
          Claus Ibsen Master

          You can use the content enricher EIP to enrich the splitted message with the file

          http://camel.apache.org/content-enricher.html

           

          But the issue is that this EIP currently does not support specifying a dynamic URI with information from the current exchange. This will be improved in the future.

           

          So what you can do instead, is just to set the message body to the file handle of the file to pickup, for example from a method call, and then use the message translator EIP and invoke the method.

           

          public File pickupFile(String name) {
            return new File(name);
          }
          

           

          Then the only question remains, should you do something with the file after you have successfully processed it? eg should you move or delete the file, or nothing at all? If you have to do something, then you would need to do this manually, for example using java code, to eg delete the file etc.

          • 2. Re: A design question
            nicolas duminil Master

            Hi Claus,

             

            Many thanks for your answer. I'm not sure I understood what you meant but I came at the following code:

             

                from("file:input").split().tokenize("\n").process(new FileConsumer()).bean(FileProcessor.class);

             

            public class FileConsumer implements Processor

            {

              private static Logger LOG = Logger.getLogger(FileConsumer.class);

               

              @Override

              public void process(Exchange exchg) throws Exception

              {

                ConsumerTemplate ct = exchg.getContext().createConsumerTemplate();

                String filePath = (String)exchg.getIn().getBody();

                Endpoint ep = exchg.getContext().getEndpoint("file:input/files?fileName=" + filePath.substring(0, filePath.length()-2));

                exchg = ct.receive(ep);

              }

            }

             

            The meta-file, ie the file containing file names, will be picked-up and splitted line by line. each line will be passed, as a string, to the FileConsumer which instantiates an endpoint and on its behalf consumes the file and passes it to the next node in the route.

             

            What do you think, is that a good design ?

             

            Kind regards,

             

            Nicolas

            • 3. Re: A design question
              Claus Ibsen Master

              Hi

               

              I think your design is a bit overkill, and its also being dependent on the Camel API.

               

              For example you could have a java bean with this code

              public class FileConsumer {
              
              public String loadFile(String filePath) {
              String path = filePath.substring(0, filePath.length()-2);
              return new File("input/files/" + path));
              }
              

               

              And then use that in the route

              from("file:input")
                .split().tokenize("\n")
                  .bean(FileConsumer.class)
                  .bean(FileProcessor.class);
              

               

              • 4. Re: A design question
                Claus Ibsen Master

                But it also depends a bit what you do in the FileProcessor? As with my code you essentially set the message body to a java.io.File handle. Where as with your code, you use the ConsumerTemplate API to poll a message (which happens to be file based).

                 

                The latter supports the various options on the file component of Camel, so you automatic can move/delete the file afterwards, where as in the first solution you will have to do this yourself.

                 

                Also ideally you should not create a new consumer template all the time. See this FAQ as it also applies for the consumer template

                http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html

                • 5. Re: A design question
                  nicolas duminil Master

                  Hi Claus,

                   

                  Yes, I understand now.

                   

                  Many thanks.

                   

                  Nicolas