6 Replies Latest reply on Nov 10, 2008 2:32 PM by Ryan Hochstetler

    FTP Large Files

    Joe Kemp Newbie

      Is it possible to FTP large files with the ftp-listener without the file ever ending up in memory? I see how to process large files using the huge-split quickstart. Does something like this exist for the ftp-listener? I want to process 500 meg files and rather not have a separate process sftp them into a directory for fs-listener.

        • 1. Re: FTP Large Files
          Daniel Bevenius Master

          Hi,

          If you modify the jboss-esb-unfiltered.xml file in the helloworld_ftp_action quickstart to use the following settings:

          <?xml version = "1.0" encoding = "UTF-8"?>
          <jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
           <providers>
           <ftp-provider name="FTPprovider" hostname="@FTP_HOSTNAME@" >
           <ftp-bus busid="helloFTPChannel" >
           <ftp-message-filter
           username="@FTP_USERNAME@"
           password="@FTP_PASSWORD@"
           passive="false"
           directory="@FTP_DIRECTORY@"
           input-suffix=".dat"
           work-suffix=".esbWorking"
           post-delete="false"
           post-suffix=".COMPLETE"
           error-delete="false"
           error-suffix=".HAS_ERROR"
           />
           </ftp-bus>
           </ftp-provider>
          
           <jms-provider name="JBossMessaging" connection-factory="ConnectionFactory">
           <jms-bus busid="quickstartEsbChannel">
           <jms-message-filter dest-type="QUEUE" dest-name="queue/quickstart_helloworld_ftp_esb"
           selector="source='fromHelloworldFTPAction'"/>
           </jms-bus>
           </jms-provider>
          
           <schedule-provider name="cronExample">
           <cron-schedule scheduleid="cron-schedule" cronExpression="0/5 * * * * ?"/>
           </schedule-provider>
          
           </providers>
           <services>
           <service
           category="myCategory"
           name="myFileListener"
           description="Hello World File Action (esb listener)" >
           <listeners>
           <ftp-listener name="FtpGateway"
           busidref="helloFTPChannel"
           is-gateway="true"
           scheduleidref="cron-schedule">
           <property name="composer-class" value="org.jboss.soa.esb.listeners.gateway.RemoteFileNameMessageComposer"/>
           </ftp-listener>
           <jms-listener name="helloWorldFileAction" busidref="quickstartEsbChannel" />
           </listeners>
           <actions mep="OneWay">
           <action name="action2" class="org.jboss.soa.esb.actions.SystemPrintln">
           <property name="printfull" value="true"/>
           </action>
           </actions>
           </service>
           </services>
          </jbossesb>
          

          If you then look at the output you will see that the path to the file being process will be in the messages body and not the files content.

          Regards,

          /Daniel

          • 2. Re: FTP Large Files
            Joe Kemp Newbie

            I'm getting the filename in the message now but the file is no where to be found. I am not sure if something is deleting the file or it is never really being created.

            • 3. Re: FTP Large Files
              Daniel Bevenius Master

              Sorry my bad here. ..

              The fs listener specifies the attribute 'post-rename' to stop the ESB from renaming it after it has processed the file. Processed in most cases consist of extracting the contents of the file, and packaging it into an ESB Message object.
              But I just tried this and noticed that the 'post-rename' attribute is not available for the ftp-message-filter.
              I'll create a JIRA for this. Sorry for misleading you

              Regards,

              /Daniel

              • 4. Re: FTP Large Files
                Kevin Conner Master

                The remote code is a specialisation of the local filesystem code so you may still be able to do this through properties.

                I haven't tried this, so take it with as much salt as you wish, but try creating a property named 'postRename' and see if it is picked up.

                • 5. Re: FTP Large Files
                  Joe Kemp Newbie

                  I guess I am not sure what is happening. post-rename would suggest the file is there but being renamed. Trying to find a work around until it gets fixed.

                  • 6. Re: FTP Large Files
                    Ryan Hochstetler Newbie

                    The way I got around this (since our implementation will pull large files) is to create an FtpLocalFileNameMessageComposer and inject it into the ftp-listener config. RemoteFileNameMessageComposer is mostly worthless because it just gives you the path of the remote file and leaves it up to the action processing pipeline to do the downloading.

                    If you look at the default MessageComposer (RemoteFileMessageCompser), it does all of the downloading in the compose() method. Just make your own composer to download the file to a local directory and then return only the location of the local file. Of course, then you must do your own file management on the local side (delete after processed, etc), but it avoids huge memory consumption.

                    I'd give you the code, but it's not mine to give away and I actually like my job, so I'm hesitant to jeopardize. It shouldn't be difficult to modify the existing composer to do your bidding.