Which version of the ESB are you using?
These files are very large. All the listeners (including the file gateway listener), by default, read the full message into memory (into the ESB normalized message). With messages/files of this size, you're going to run into trouble running the ESB out-of-the-box.
One way to solve it in the current codebase is to implement a custom "MessageComposer" and configure it on the file listener. What this composer will do depends on which ESB codebase you're using.
4.2.x Release Distro: The custom composer impl would compose an ESB message containing a String payload with the name of the file. You action pipeline would then need actions to open the file stream etc. This is a flaw in the 4.2.x design, but it has been fixed in source.
SVN - branches/JBESB_4_2_1_GA_FP1: The custom composer impl would compose an ESB message and in its payload, set an InputStream to the file in question. Using the inVM transport, this message can now be pushed into the connected action pipeline wheere it can be handled in a streaming fashion.
I am using version 4.2. Also, my message does not include the contents of those large files. The large files are always accompanied by their associated metadata files (which range anywhere from 2k - 3k bytes). It is those metadata files I am monitoring for in my incoming directory and as such the message includes the content of those metadata files (and not the content from actual data file). From that metadata, I retreive the actual data file (those large files) and run the checksum operation on them from my EJB.
Again, I am looking to increase the throughput where I can generate checksum on as many files as possible in any given run time.