Lots of txt files to read

[lex32]

Hi,

as I am having the demand to read > 2000 text files a day, there are the following requirements:

- Each textfile is 200k -> 900k

- textfiles are structured row-based; similar to EDI

- fields and data have to be check on validation

- no file / no line is allowed to be lost

- only 2,5 hours time for processing

- within the files are turnovers, these have to be processed at first

To manage this we were thinking about using Active MQ to insure consitency and then perhaps call Kettle on x diffrent stations in order to do load balancing.

Any ideas ?

[davestanley]

In terms of the broker, it can provide you with various qualities of service to ensure your messaging layer is reliable. You would probably want any messages you send to the broker be PERSISTENT.

See this section of the Fuse Doc's for more info.

Also, you could use transacted sessions, such that all the work in a session can be rolledback on an error.

Its hard to tell from your description if ActiveMQ alone will give you all you need.  Camel/Fuse Mediation Router might also be worth investigating. You maybe able to use a camel route which would suck in the files, then pass them on to activemq and from there to a processing stage (using kettle in a bean processor). (ActiveMQ supports embedding camel routes via ActiveMQ.xml)

Hope this helps

/Dave