Parallel processing

[richardmoore]

I have a recurring design pattern I was hoping to get some help with the best way to approach the problem.

We have 11 facilities and have to run processing and/or extracts of data for all of the facilities. We brought on the last 2 new facilities just recently and all of the jobs required a change to account for the new facilities. Is there a way to run in parallel for a changing list of facilities as new ones are brought on without making changes to the jobs?

[cfang]

looks like you are looking for a smart item reader that can dynamically detect the collection of input resources, and also dynamically assign them to chunk partitioins.  One simple way is to deposit all facility data into a shared location / directory, and assign each facility data file to its own partition.  You can implement javax.batch.api.chunk.CheckpointAlgorithm to dynamically assign partitions.

If each data file is not big enough to warranty its own partition, you can assign multiple input files to a partition.  We currently have a open feature request to implement an item reader that reads from multiple resource:

1. JBERET-314

Implement ItemReader that reads from multiple resources