Suggestions to improve my ESB app performance

[jakkur]

Hi,
I am looking for suggestions to improve the performance of my prototype ESB application. Briefly, the app consists of a couple of services. The first one receives messages from FileSystem provider which monitors a directory for incoming metadata files (these and their associated data files come ~ at the rate of 7/sec, and average size of the data file is about 250MB). After the first service receives the message, the action within that service performs some basic validations on the data file. It then invokes second service (ServiceInvoker) which implements an action that invokes an EJB that generates an MD5 checksum for each data file. The subsequent action within the second service then persist the checksum and other metadata of the file into DB.

When I deploy my app into the ESB server and trigger it by dropping the files in the incomig directory, the EJB approximately is taking an average of about 10 secs to genertate the checksum on each data file. So, If drop 10 data files of 300MB each (along with their associated metadata files), the application is processing 10 messages and takes about 100 secs before it finsihes generating checksums on those. While I have no choice/control on the time it is taking to generate checksum on any individual file (command line MD5 checksum programs take almost same or little less time on the same file), I want to speed up the whole process by means such as running the checksums in parallel (by controlling maxThreads attribute of the jms-listener?) or deploying the ejb in another instance of application server different from the one where my ESB server is running or simply in another JVM, etc.

I have a requirement, for example, that my app should perform the above described scenario in less than half the time it is taking now. I have already in included the typical jre optimization stuff in the above scenario. I am looking for something that JBoss ESB/AS can provide me to speed the whole prcesses.

Thanks,
Jakku

[tfennelly]

Hi there.

Which version of the ESB are you using?

These files are very large. All the listeners (including the file gateway listener), by default, read the full message into memory (into the ESB normalized message). With messages/files of this size, you're going to run into trouble running the ESB out-of-the-box.

One way to solve it in the current codebase is to implement a custom "MessageComposer" and configure it on the file listener. What this composer will do depends on which ESB codebase you're using.

4.2.x Release Distro: The custom composer impl would compose an ESB message containing a String payload with the name of the file. You action pipeline would then need actions to open the file stream etc. This is a flaw in the 4.2.x design, but it has been fixed in source.

SVN - branches/JBESB_4_2_1_GA_FP1: The custom composer impl would compose an ESB message and in its payload, set an InputStream to the file in question. Using the inVM transport, this message can now be pushed into the connected action pipeline wheere it can be handled in a streaming fashion.

HTH.

[jakkur]

I am using version 4.2. Also, my message does not include the contents of those large files. The large files are always accompanied by their associated metadata files (which range anywhere from 2k - 3k bytes). It is those metadata files I am monitoring for in my incoming directory and as such the message includes the content of those metadata files (and not the content from actual data file). From that metadata, I retreive the actual data file (those large files) and run the checksum operation on them from my EJB.

Again, I am looking to increase the throughput where I can generate checksum on as many files as possible in any given run time.

Thanks.
Jakku