6 Replies Latest reply on May 12, 2009 1:27 AM by bate

    How to create a custom FTPProvider for large file transfer

    Todd Nash Newbie

      We am trying to prove out the use of JBossESB over Mule. One of the criteria that we have to meet is the ability to ftp a large file from a remote FTP site to a local filesystem. Then we can process the file.

      Looking at the samples included, the FTP samples all encapsulate the file in a Message and send it to a JMS listener. This won't work for our situation. We get customer data files in our specified format that is similar, but not the same as EDI. These files can be upwards of 1GB in size. We need an FTP provider that can write the file to a shared filesystem locally and send a message containing the name of the file.

      How can we create our own implementation of FsProviderDocument.FsProvider? I have looked through the source code, but I can't find the source for FsProviderDocumentImpl.FsProviderImpl.

      Or is this not the preferred way to implement a custom provider. Should we be creating a custom Gateway using AbstractThreadedManagedLifecycle?

      My timeframe for this POC is short, I am trying to prove the use of JBoss over Mule, so any help would be appreciated.

        • 1. Re: How to create a custom FTPProvider for large file transf
          bate Newbie

          i had the same probleme 2 month ago.
          i solved it by extending the class org.jboss.soa.esb.listeners.gateway.RemoteFileMessageComposer
          i had to modify in particular the methode
          private byte[] getFileContents(File file) throws IOException, RemoteFileSystemException . Here is where the content of the file is beeing streamed over to the wire. you can copy the file to a file into the local file system. this is exactly what i did. i wrote not more thank 5 lines of codes.
          My probleme was how to get the name of the original file.
          If you are interested i may send you my overriden file for RemoteFileMessageComposer file

          /Kaba

          • 2. Re: How to create a custom FTPProvider for large file transf
            Todd Nash Newbie

            Kaba,

            Any help would be welcomed. How did you access your extended class? Did you rebuild the jboss jar file?

            Thanks,

            Todd

            • 3. Re: How to create a custom FTPProvider for large file transf
              bate Newbie

              No i just downloaded the RemoteFileComposer and extended it. i compiled only one file , the one io wrote. Just download the RemoteFileComposer to the samples\quickstarts\helloworld_ftp_action\src\org\jboss\soa\esb\samples\quickstart\helloworldftpaction directory.
              and the


              package org.jboss.soa.esb.listeners.gateway;

              import org.jboss.soa.esb.listeners.message.MessageComposer;
              import org.jboss.soa.esb.listeners.message.MessageDeliverException;
              import org.jboss.soa.esb.listeners.ListenerUtil;
              import org.jboss.soa.esb.listeners.ListenerTagNames;
              import org.jboss.soa.esb.helpers.ConfigTree;
              import org.jboss.soa.esb.ConfigurationException;
              import org.jboss.soa.esb.addressing.EPR;
              import org.jboss.soa.esb.addressing.eprs.FTPEpr;
              import org.jboss.soa.esb.util.*;
              import org.jboss.soa.esb.message.Message;
              import org.jboss.soa.esb.message.MessagePayloadProxy;
              import org.jboss.soa.esb.message.body.content.BytesBody;
              import org.jboss.soa.esb.message.format.MessageFactory;
              import org.jboss.internal.soa.esb.assertion.AssertArgument;

              import java.io.File;
              import java.io.FileInputStream;
              import java.io.FileOutputStream;
              import java.io.IOException;
              import java.io.InputStream;
              import java.nio.channels.FileChannel;
              import java.util.Properties;
              /**
              * Remote File Message Composer.
              * <p/>
              * The logic in this class was lifeted from {@link RemoteGatewayListener}.
              *
              * @see org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer
              * @author <a href="mailto:tom.fennelly@jboss.com">tom.fennelly@jboss.com</a>
              */
              public class MyRemoteFileMessageComposer<T extends File> implements MessageComposer<T> {

              private FTPEpr ftpEpr;
              private File downloadDir;
              private MessagePayloadProxy payloadProxy;
              private String out_dir;

              public void setConfiguration(ConfigTree config) throws ConfigurationException {
              String output_dir = config.getAttribute(ListenerTagNames.FILE_POST_DIR_TAG);

              String properti=System.getProperty("ftp.properties.file");
              System.out.println("ftp.properties.file"+properti);
              try
              {

              FileInputStream propFile = new FileInputStream(properti);
              Properties p = new Properties(System.getProperties());
              p.load(propFile);
              System.setProperties(p);
              out_dir=System.getProperty("jbossesb.ftp.directory");

              }catch (Exception e)
              {

              e.printStackTrace();
              }



              System.out.println("Getting*"+output_dir+"jbossesb.ftp.directory:"+System.getProperty("jbossesb.ftp.directory"));
              EPR epr = ListenerUtil.assembleEpr(config);

              if (!(epr instanceof FTPEpr)) {
              throw new ConfigurationException("This Gateway only accepts FTP and SFTP.");
              }
              ftpEpr = (FTPEpr) epr;
              // This may look a bit odd... The "file" input dir is our FTP download dir...
              downloadDir = AbstractFileGateway.getFileInputDirectory(config);
              payloadProxy = new MessagePayloadProxy(config,
              new String[] {BytesBody.BYTES_LOCATION},
              new String[] {BytesBody.BYTES_LOCATION});
              }

              public Message compose(T inputFile) throws MessageDeliverException {
              AssertArgument.isNotNull(inputFile, "inputFile");

              Message message = MessageFactory.getInstance().getMessage();
              System.out.println("Setting******************************************************");
              message.getProperties().setProperty("FILE_ABSOLUTE_PATH",inputFile.getAbsolutePath());
              message.getProperties().setProperty("FILE_NAME",inputFile.getName());
              System.out.println("output dir: " + System.getProperty("output.directory")+"*"+System.getProperty("jboss.home")+"***********"+message.getProperties().getProperty("FILE_ABSOLUTE_PATH")+ "NAME:"+message.getProperties().getProperty("FILE_NAME"));

              try {
              payloadProxy.setPayload(message, getPayload(inputFile));
              } catch (IOException e) {
              throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
              } catch (RemoteFileSystemException e) {
              throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
              }

              return message;
              }

              protected Object getPayload( T inputFile ) throws RemoteFileSystemException, IOException {
              return getFileContents(inputFile);
              }

              public Object decompose(Message message, T inputMessage) throws MessageDeliverException {
              return payloadProxy.getPayload(message);
              }

              private byte[] getFileContents(File file) throws IOException, RemoteFileSystemException {
              RemoteFileSystem remoteFileSystem = null;
              File temp = File.createTempFile("FTPdown", ".tmp");
              byte[] returnBytes;

              try
              {

              remoteFileSystem = RemoteFileSystemFactory.getRemoteFileSystem(ftpEpr, true);
              remoteFileSystem.setRemoteDir(FtpClientUtil.fileToFtpString(downloadDir));
              remoteFileSystem.downloadFile(file.toString(), temp.getAbsolutePath());
              returnBytes=FileUtil.readFile(temp);

              File target= new File(out_dir+"/"+file.getName().substring(0,file.getName().lastIndexOf(".")));

              FileChannel in = (new FileInputStream(temp)).getChannel();
              FileChannel out = (new FileOutputStream(target)).getChannel();
              in.transferTo(0, temp.length(), out);
              in.close();
              out.close();

              return returnBytes;
              } finally {
              // File target= new File("/tmp/prop/"+file.getName().substring(0,file.getName().lastIndexOf(".")));
              //FileChannel in = (new FileInputStream(temp)).getChannel();
              //FileChannel out = (new FileOutputStream(target)).getChannel();
              //in.transferTo(0, temp.length(), out);
              //in.close();
              // out.close();

              // temp.renameTo(new File("/tmp/prop/"+file.getName().substring(0,file.getName().lastIndexOf("."))));

              // temp.delete();
              if (null != remoteFileSystem) {
              remoteFileSystem.quit();
              }
              }
              }
              }



              my jboss-esb.xml was like this

              <?xml version = "1.0" encoding = "UTF-8"?>



              <ftp-provider name="FTPprovider" hostname="yourHostname:22" >
              <ftp-bus busid="helloFTPChannel" >
              <ftp-message-filter
              protocol="sftp"
              username="yourusername"
              password="youruserpasswd"
              passive="true"
              directory="/localdata/in_arbeit"
              post-directory="/localdata/kopiert"
              error-directory="/localdata/Fehler"
              input-suffix=".pdf"
              work-suffix=".esbWorking"
              post-delete="false"
              post-suffix=".COMPLETE"
              error-delete="false"
              error-suffix=".HAS_ERROR"
              />

              </ftp-bus>
              </ftp-provider>

              <jms-provider name="JBossMQ" connection-factory="ConnectionFactory">

              <jms-bus busid="quickstartEsbChannel">
              <jms-message-filter
              dest-type="QUEUE"
              dest-name="queue/quickstart_helloworld_ftp_esb"
              selector="source='fromHelloworldFTPAction'"
              />
              </jms-bus>

              </jms-provider>

              <schedule-provider name="cronExample">
              <cron-schedule scheduleid="cron-schedule" cronExpression="0/5 * * * * ?"/>
              </schedule-provider>






              <ftp-listener name="FtpGateway"
              busidref="helloFTPChannel"
              is-gateway="true"
              scheduleidref="cron-schedule">

              </ftp-listener>
              <jms-listener name="helloWorldFileAction"
              busidref="quickstartEsbChannel"
              />











              • 4. Re: How to create a custom FTPProvider for large file transf
                bate Newbie

                Please note you will have to pass the property "ftp.properties.file" pointing the the fully qualified filename using i.e the -Dftp.properties.file option of jvm while starting JBoss. I just modified the run.conf in bin directory of jboss Server.
                In this file you will dfifine the property jbossesb.ftp.directory pointing the the share diretory to copy the remote file into localy
                Good luck
                /mady

                • 5. Re: How to create a custom FTPProvider for large file transf
                  Todd Nash Newbie

                  Ok, I built everything like you explained, but I can't get the ESB to see my implementation of RemoteFileMessageComposer. Did you do something to get the classloader to load your version over the version in the sar?

                  • 6. Re: How to create a custom FTPProvider for large file transf
                    bate Newbie

                    this much more simpler. actually i tried to copy my whole jboss-esb.xml but it was cut.
                    to pick your implementation of the MessageCompser say org.jboss.soa.esb.listeners.gateway.MYFileMessageComposer you just have to declare it in your jboss-esb.xml file unter the listener section.
                    You will have something like this:
                    Please note this < is missing just before the property i cant get jboss forum tools to post my text as it should be.
                    i had tried many times to post my jboss-esb.xml


                    <ftp-listener name="FtpGateway"
                    busidref="digipenFTPChannel"
                    is-gateway="true"
                    scheduleidref="cron-schedule">
                    property name="composer-class" value="org.jboss.soa.esb.listeners.gateway.MYFileMessageComposer"/>
                    </ftp-listener>
                    <jms-listener name="digipenJMSListener"
                    busidref="digipen-ESBChannel"
                    />