I trying to determine whether or not an ESB is the correct model for our use csae. We have the need to process files that arrive on an FTP server. The contents of the files are formatted in a structure similar to EDI, but not EDI. The files are typically 500 meg to 2GB in size. The first record in the file determines how the file is processed.
Each record in the file is translated based on some business rules and persisted to the database.
Users want to be notified when a file arrives on the FTP server, when the file validation fails, how many records are persisted to the database.
We also have the need to service external requests for access to the data from web service calls.
In addition, the opposite use case exists where the data is exported from the database and translated to the EDI like file and sent to an FTP server.
Does this sound like a good use for an ESB? Much of my consternation comes from the fact that ESBs work on Messages and Messages are limited in size.
I have looked into using the FTP provider to move a file to a local filesystem and Message the filename to the bus. However, the implementations of the JBoss FTP provider doesn't handle streams and can't be configured OOTB to move a file to a filesytem. Putting the entire file contents in the Message causes memory problems.
Correct me if I am wrong, but it appears the ESBs are limited to messaging small content "packages".
Any feedback would be appreciated.
