FTP x number of files at 1 time

[chuaky]

hi,

Got a query on FTP listener, please advise.

My setup is to use FTP listener to retrieve XML files from a FTP server and then do some processing on it. As per standard, the FTP file would be send via JMS for processing in the actions list.

The JMS was configured to use postgres DB , because i encountered heap memory overflow when using the default hsql. After using postgres, the problem is less sever but can still happen. The transfer file was big (approximately 10M each).

I also notice that FTP listener "getFileList" take all the files in the remote FTP server each time. Would it be possible to specify we only want to retrieve 4 files on each FTP scheduler. I think that may help to distribute the load more evener and less memory stress on JMS persistence layer.

By the way, this situation occurs when i have many backlog files at the remote ftp server to process, and 1 time download of 96 big files can be very stress to java memory :(

Hope there is a way to control the FTP listener to retrieve x number of files at each schedule timeout.

Thank you.

[tfennelly]

I looked at the code and config and I don't see a way of filtering the remote file list in this way.

In any case, it sounds like your #1 issue is the processing of these 10M files. You should really be splitting these up before routing them into the ESB. Sucking 10M into memory and JMS/DB for every message is a recipe for disaster :-( See the "huge-split-...." quickstart.

[chuaky]

hi tfennelly,

The huge file example use a fs-listener example, i am not sure if it applies to ftp. Processing the 10M file takes less than 20 seconds, for a single file processing (including JMS writing to DB).

There is a for loop for the filelist, maybe could abort is the count exceeds a threshold set in the ftp-message-filter ???

Thanks

package org.jboss.soa.esb.listeners.gateway;
...
public abstract class AbstractFileGateway extends AbstractManagedLifecycle implements ScheduledEventListener {

 /**
 * Execute on trigger from the scheduler.
 */
 public void onSchedule() throws SchedulingException {

 File[] fileList;
 try {
 fileList = getFileList();
 if(fileList == null) {
 _logger.warn("No files to process.");
 return;
 }
 }
 catch (GatewayException e) {
 _logger.error("Can't retrieve file list", e);
 return;
 }

 for (File fileIn : fileList) {

[tfennelly]

"chuaky" wrote:
The huge file example use a fs-listener example, i am not sure if it applies to ftp. Processing the 10M file takes less than 20 seconds, for a single file processing (including JMS writing to DB).

Sure it can process a single 10M file, but can it handle 10 or 100 of them (concurrently)?

You should be able to modify the fs-listener to work for FTP. If you have large files (10M is large), this is the only reliable way of processing them.

"chuaky" wrote:
There is a for loop for the filelist, maybe could abort is the count exceeds a threshold set in the ftp-message-filter ???

Sure. Give it a try if you're happy with that.

[chuaky]

hi tfennelly,

Thanks for the feedback.

Sure it can process a single 10M file, but can it handle 10 or 100 of them (concurrently)?

It can't processing 100 of them at same time, because the JMS is very busy writing into DB. Maybe controlling the rate ESB is receiving the FTP files can resolve the issue.

Best regards.