8 Replies Latest reply on Dec 11, 2008 4:58 AM by bate

    Copying a file from sftp Server to a fileshare

    bate Newbie

      Hello,
      do i have to chain a ftp-provider and fs-provider in order to copy a file from sftp Server to a fileshare? or i can do this by just configuring properly a ftp-provider
      thank

        • 1. Re: Copying a file from sftp Server to a fileshare
          Kevin Conner Master

          The gateways will pull the message down and create an ESB message from it.

          What are you trying to achieve?

          • 2. Re: Copying a file from sftp Server to a fileshare
            bate Newbie

            i want to copy a file from a directory on a sftp server to another directory on another server.

            ich have just configured the directry to monitor on the sftp server.
            The file is beeing picked up and copied to and processed.
            But i am getting the following exception from the jboss server log:


            CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
            INET/inet_error: read errno = 10054

            CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
            INET/inet_error: receive in try_connect errno = 203

            CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
            INET/inet_error: read errno = 10054

            CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
            INET/inet_error: receive in try_connect errno = 203

            CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
            INET/inet_error: read errno = 10054

            CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
            INET/inet_error: receive in try_connect errno = 203

            CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
            INET/inet_error: read errno = 10054

            CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
            INET/inet_error: receive in try_connect errno = 203

            CBEM35NTB7403 (Client) Mon Nov 24 01:00:57 2008
            INET/INET_connect: gethostbyname failed, error code = 11001

            CBEM35NTB7403 (Client) Mon Nov 24 01:01:02 2008
            INET/INET_connect: gethostbyname failed, error code = 11001

            CBEM35NTB7403 (Client) Mon Nov 24 01:01:04 2008
            INET/INET_connect: gethostbyname failed, error code = 11001

            CBEM35NTB7403 (Client) Mon Nov 24 01:01:09 2008
            INET/INET_connect: gethostbyname failed, error code = 11001

            CBEM35NTB7403 (Client) Mon Nov 24 01:02:01 2008
            INET/INET_connect: gethostbyname failed, error code = 11001

            CBEM35NTB7403 (Client) Mon Nov 24 01:02:05 2008
            INET/INET_connect: gethostbyname failed, error code = 11001



            • 3. Re: Copying a file from sftp Server to a fileshare
              bate Newbie

               

              "kaba" wrote:
              i want to copy a file from a directory on a sftp server to another directory on another server.

              ich have just configured the directry to monitor on the sftp server.
              The file is beeing picked up and copied to and processed.
              But i am getting the following exception from the jboss server log:


              CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
              INET/inet_error: read errno = 10054

              CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
              INET/inet_error: receive in try_connect errno = 203

              CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
              INET/inet_error: read errno = 10054

              CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
              INET/inet_error: receive in try_connect errno = 203

              CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
              INET/inet_error: read errno = 10054

              CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
              INET/inet_error: receive in try_connect errno = 203

              CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
              INET/inet_error: read errno = 10054

              CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
              INET/inet_error: receive in try_connect errno = 203

              CBEM35NTB7403 (Client) Mon Nov 24 01:00:57 2008
              INET/INET_connect: gethostbyname failed, error code = 11001

              CBEM35NTB7403 (Client) Mon Nov 24 01:01:02 2008
              INET/INET_connect: gethostbyname failed, error code = 11001

              CBEM35NTB7403 (Client) Mon Nov 24 01:01:04 2008
              INET/INET_connect: gethostbyname failed, error code = 11001

              CBEM35NTB7403 (Client) Mon Nov 24 01:01:09 2008
              INET/INET_connect: gethostbyname failed, error code = 11001

              CBEM35NTB7403 (Client) Mon Nov 24 01:02:01 2008
              INET/INET_connect: gethostbyname failed, error code = 11001

              CBEM35NTB7403 (Client) Mon Nov 24 01:02:05 2008
              INET/INET_connect: gethostbyname failed, error code = 11001



              is looks like there is a network problem. But i can transfer file using ftp from shell


              sftp> put test.dat
              Uploading test.dat to /localdata/Eingang/test.dat
              test.dat 100% 5 0.0KB/s 00:00
              sftp> ls
              bird.dat.esbWorking test.dat.esbWorking
              sftp> ls
              bird.dat.esbWorking
              sftp> rm bird.dat.*
              Removing /localdata/Eingang/bird.dat.esbWorking
              sftp> ls
              sftp> ls ../Abgeschlossen
              ../Abgeschlossen/bird.dat.COMPLETE ../Abgeschlossen/test.dat.COMPLETE
              sftp> get test.dat.COMPLETE
              Couldn't stat remote file: No such file or directory
              File "/localdata/Eingang/test.dat.COMPLETE" not found.
              sftp> ls
              sftp> pwd
              Remote working directory: /localdata/Eingang
              sftp> get ../Abgeschlossen/test.dat.COMPLETE
              Fetching /localdata/Eingang/../Abgeschlossen/test.dat.COMPLETE to test.dat.COMPLETE
              /localdata/Eingang/../Abgeschlossen/test.dat.COMPLETE 100% 5 0.0KB/s 00:00
              sftp>



              • 4. Re: Copying a file from sftp Server to a fileshare
                bate Newbie

                Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
                Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
                Thank you for your advice


                • 5. Re: Copying a file from sftp Server to a fileshare
                  Ryan Hochstetler Newbie

                   

                  "kaba" wrote:
                  Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
                  Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
                  Thank you for your advice




                  The FTP Listener will provide the file contents as a byte array in the body of an ESB message. Yes, you must write an action to write out those bytes to disk.

                  Alternately, you can
                  1.) Use RemoteFileNameMessageComposer.java which just returns the result of remoteFile.getAbsolutePath() in the body and download the file yourself
                  or
                  2.) Write your own remote file message composer that satisfies your requirements. I suggest you begin by extending http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/rosetta/src/org/jboss/soa/esb/listeners/gateway/RemoteFileMessageComposer.java. Perhaps your composer should download the file to a tmp directory and provide the location of the downloaded file on the ESB message body. Then you could write an action to move it to the desired location.

                  Specify your message composer by setting the composer-class property inside the ftp-listener element to the fully-qualified class name. See quickstart:huge-split-enrich-transform-route for an example of setting the composer-class.

                  • 6. Re: Copying a file from sftp Server to a fileshare
                    bate Newbie

                    thank you ryanhos very much.
                    i am tryying to implement the 2. option using the RemoteFileMessageComposer. i have added the following property

                    property name="composer-class" value="org.jboss.soa.esb.listeners.gateway.RemoteFileMessageComposer"


                    But i cant see where to configure the download directory.
                    Where to set the directory to download the file to?
                    And how to get the name of the file beeing downloaded?
                    Do i have to chain with the RemoteFileNameMessageComposer class?
                    if so what is the name of this property?
                    Thank you

                    • 7. Re: Copying a file from sftp Server to a fileshare
                      bate Newbie

                       


                      "kaba" wrote:
                      Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
                      Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
                      Thank you for your advice




                      The FTP Listener will provide the file contents as a byte array in the body of an ESB message. Yes, you must write an action to write out those bytes to disk.

                      Alternately, you can
                      1.) Use RemoteFileNameMessageComposer.java which just returns the result of remoteFile.getAbsolutePath() in the body and download the file yourself

                      or
                      2.) Write your own remote file message composer that satisfies your requirements. I suggest you begin by extending http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/rosetta/src/org/jboss/soa/esb/listeners/gateway/RemoteFileMessageComposer.java . Perhaps your composer should download the file to a tmp directory and provide the location of the downloaded file on the ESB message body. Then you could write an action to move it to the desired location.

                      Specify your message composer by setting the composer-class property inside the ftp-listener element to the fully-qualified class name. See quickstart:huge-split-enrich-transform-route for an example of setting the composer-class.


                      for the 1.)
                      Is there any API in JbossESB i can use to download? or i need an external tools for this?
                      for the 2.)
                      i have try this but i do no see any effekt.
                      I would prefer using an action to get the bytes stream and write to a file on disk. This works well the only problem i have now is how to get the name of the file to download in an action?
                      Your help is very much ch appreciated


                      • 8. Re: Copying a file from sftp Server to a fileshare
                        bate Newbie

                        i managed to find a way to download the file to a local disk by modifying the RemoteFileMessageComposer.java class as fo lows: See the bold highlighted section



                        package org.jboss.soa.esb.listeners.gateway;

                        import org.jboss.soa.esb.listeners.message.MessageComposer;
                        import org.jboss.soa.esb.listeners.message.MessageDeliverException;
                        import org.jboss.soa.esb.listeners.ListenerUtil;
                        import org.jboss.soa.esb.listeners.ListenerTagNames;
                        import org.jboss.soa.esb.helpers.ConfigTree;
                        import org.jboss.soa.esb.ConfigurationException;
                        import org.jboss.soa.esb.addressing.EPR;
                        import org.jboss.soa.esb.addressing.eprs.FTPEpr;
                        import org.jboss.soa.esb.util.*;
                        import org.jboss.soa.esb.message.Message;
                        import org.jboss.soa.esb.message.MessagePayloadProxy;
                        import org.jboss.soa.esb.message.body.content.BytesBody;
                        import org.jboss.soa.esb.message.format.MessageFactory;
                        import org.jboss.internal.soa.esb.assertion.AssertArgument;

                        import java.io.File;
                        import java.io.FileInputStream;
                        import java.io.FileOutputStream;
                        import java.io.IOException;
                        import java.io.InputStream;
                        import java.nio.channels.FileChannel;
                        /**
                        * Remote File Message Composer.
                        * <p/>
                        * The logic in this class was lifeted from {@link RemoteGatewayListener}.
                        *
                        * @see org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer
                        * @author <a href="mailto:tom.fennelly@jboss.com">tom.fennelly@jboss.com</a>
                        */
                        public class BZPARemoteFileMessageComposer<T extends File> implements MessageComposer<T> {

                        private FTPEpr ftpEpr;
                        private File downloadDir;
                        private MessagePayloadProxy payloadProxy;

                        public void setConfiguration(ConfigTree config) throws ConfigurationException {
                        String output_dir = config.getAttribute("output_dir");
                        System.out.println("Getting******************************************************"+output_dir);
                        EPR epr = ListenerUtil.assembleEpr(config);

                        if (!(epr instanceof FTPEpr)) {
                        throw new ConfigurationException("This Gateway only accepts FTP and SFTP.");
                        }
                        ftpEpr = (FTPEpr) epr;
                        // This may look a bit odd... The "file" input dir is our FTP download dir...
                        downloadDir = AbstractFileGateway.getFileInputDirectory(config);
                        payloadProxy = new MessagePayloadProxy(config,
                        new String[] {BytesBody.BYTES_LOCATION},
                        new String[] {BytesBody.BYTES_LOCATION});
                        }

                        public Message compose(T inputFile) throws MessageDeliverException {
                        AssertArgument.isNotNull(inputFile, "inputFile");

                        Message message = MessageFactory.getInstance().getMessage();
                        System.out.println("Setting******************************************************");
                        message.getProperties().setProperty("FILE_ABSOLUTE_PATH",inputFile.getAbsolutePath());
                        message.getProperties().setProperty("FILE_NAME",inputFile.getName());
                        System.out.println("Setting******************************************************"+message.getProperties().getProperty("FILE_ABSOLUTE_PATH")+ "NAME:"+message.getProperties().getProperty("FILE_NAME"));
                        try {
                        payloadProxy.setPayload(message, getPayload(inputFile));
                        } catch (IOException e) {
                        throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
                        } catch (RemoteFileSystemException e) {
                        throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
                        }

                        return message;
                        }

                        protected Object getPayload( T inputFile ) throws RemoteFileSystemException, IOException {
                        return getFileContents(inputFile);
                        }

                        public Object decompose(Message message, T inputMessage) throws MessageDeliverException {
                        return payloadProxy.getPayload(message);
                        }

                        private byte[] getFileContents(File file) throws IOException, RemoteFileSystemException {
                        RemoteFileSystem remoteFileSystem = null;
                        File temp = File.createTempFile("FTPdown", ".tmp");
                        byte[] returnBytes;
                        try
                        {
                        remoteFileSystem = RemoteFileSystemFactory.getRemoteFileSystem(ftpEpr, true);
                        remoteFileSystem.setRemoteDir(FtpClientUtil.fileToFtpString(downloadDir));
                        remoteFileSystem.downloadFile(file.toString(), temp.getAbsolutePath());

                        returnBytes=FileUtil.readFile(temp);

                        File target= new File("/tmp/prop/"+file.getName().substring(0,file.getName().lastIndexOf(".")));

                        FileChannel in = (new FileInputStream(temp)).getChannel();
                        FileChannel out = (new FileOutputStream(target)).getChannel();
                        in.transferTo(0, temp.length(), out);
                        in.close();
                        out.close();

                        return returnBytes;

                        } finally {

                        temp.delete();
                        if (null != remoteFileSystem) {
                        remoteFileSystem.quit();
                        }
                        }
                        }
                        }



                        as you can see i still have to find a way to add the output_dir to the ftp message filter or to the ftp-provider config in order not to hardcode the path "/tmp/prop/" i wonder how?
                        Thank you for your Help