Copying a file from sftp Server to a fileshare

[kaba]

Hello,
do i have to chain a ftp-provider and fs-provider in order to copy a file from sftp Server to a fileshare? or i can do this by just configuring properly a ftp-provider
thank

[kconner]

The gateways will pull the message down and create an ESB message from it.

What are you trying to achieve?

[kaba]

i want to copy a file from a directory on a sftp server to another directory on another server.

ich have just configured the directry to monitor on the sftp server.
The file is beeing picked up and copied to and processed.
But i am getting the following exception from the jboss server log:

CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Mon Nov 24 01:00:57 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:02 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:04 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:09 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:02:01 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:02:05 2008
INET/INET_connect: gethostbyname failed, error code = 11001

[kaba]

"kaba" wrote:
i want to copy a file from a directory on a sftp server to another directory on another server.

ich have just configured the directry to monitor on the sftp server.
The file is beeing picked up and copied to and processed.
But i am getting the following exception from the jboss server log:

CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:37 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:39 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:01:58 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
INET/inet_error: read errno = 10054

CBEM35NTB7403 (Client) Thu Nov 13 09:02:00 2008
INET/inet_error: receive in try_connect errno = 203

CBEM35NTB7403 (Client) Mon Nov 24 01:00:57 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:02 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:04 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:01:09 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:02:01 2008
INET/INET_connect: gethostbyname failed, error code = 11001

CBEM35NTB7403 (Client) Mon Nov 24 01:02:05 2008
INET/INET_connect: gethostbyname failed, error code = 11001

is looks like there is a network problem. But i can transfer file using ftp from shell

sftp> put test.dat
Uploading test.dat to /localdata/Eingang/test.dat
test.dat 100% 5 0.0KB/s 00:00
sftp> ls
bird.dat.esbWorking test.dat.esbWorking
sftp> ls
bird.dat.esbWorking
sftp> rm bird.dat.*
Removing /localdata/Eingang/bird.dat.esbWorking
sftp> ls
sftp> ls ../Abgeschlossen
../Abgeschlossen/bird.dat.COMPLETE ../Abgeschlossen/test.dat.COMPLETE
sftp> get test.dat.COMPLETE
Couldn't stat remote file: No such file or directory
File "/localdata/Eingang/test.dat.COMPLETE" not found.
sftp> ls
sftp> pwd
Remote working directory: /localdata/Eingang
sftp> get ../Abgeschlossen/test.dat.COMPLETE
Fetching /localdata/Eingang/../Abgeschlossen/test.dat.COMPLETE to test.dat.COMPLETE
/localdata/Eingang/../Abgeschlossen/test.dat.COMPLETE 100% 5 0.0KB/s 00:00
sftp>

[kaba]

Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
Thank you for your advice

[ryanhos]

"kaba" wrote:
Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
Thank you for your advice

The FTP Listener will provide the file contents as a byte array in the body of an ESB message. Yes, you must write an action to write out those bytes to disk.

Alternately, you can
1.) Use RemoteFileNameMessageComposer.java which just returns the result of remoteFile.getAbsolutePath() in the body and download the file yourself
or
2.) Write your own remote file message composer that satisfies your requirements. I suggest you begin by extending http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/rosetta/src/org/jboss/soa/esb/listeners/gateway/RemoteFileMessageComposer.java. Perhaps your composer should download the file to a tmp directory and provide the location of the downloaded file on the ESB message body. Then you could write an action to move it to the desired location.

Specify your message composer by setting the composer-class property inside the ftp-listener element to the fully-qualified class name. See quickstart:huge-split-enrich-transform-route for an example of setting the composer-class.

[kaba]

thank you ryanhos very much.
i am tryying to implement the 2. option using the RemoteFileMessageComposer. i have added the following property

property name="composer-class" value="org.jboss.soa.esb.listeners.gateway.RemoteFileMessageComposer"

But i cant see where to configure the download directory.
Where to set the directory to download the file to?
And how to get the name of the file beeing downloaded?
Do i have to chain with the RemoteFileNameMessageComposer class?
if so what is the name of this property?
Thank you

[kaba]

"kaba" wrote:
Do i have to write an action to catch and write the bytes stream i get from a sftp listener/provider to a file on the file system?
Is there any other convenient way of copying a file per stfp from a sftp server to a file share?
Thank you for your advice




The FTP Listener will provide the file contents as a byte array in the body of an ESB message. Yes, you must write an action to write out those bytes to disk.

Alternately, you can
1.) Use RemoteFileNameMessageComposer.java which just returns the result of remoteFile.getAbsolutePath() in the body and download the file yourself

or
2.) Write your own remote file message composer that satisfies your requirements. I suggest you begin by extending http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/rosetta/src/org/jboss/soa/esb/listeners/gateway/RemoteFileMessageComposer.java . Perhaps your composer should download the file to a tmp directory and provide the location of the downloaded file on the ESB message body. Then you could write an action to move it to the desired location.

Specify your message composer by setting the composer-class property inside the ftp-listener element to the fully-qualified class name. See quickstart:huge-split-enrich-transform-route for an example of setting the composer-class.

for the 1.)
Is there any API in JbossESB i can use to download? or i need an external tools for this?
for the 2.)
i have try this but i do no see any effekt.
I would prefer using an action to get the bytes stream and write to a file on disk. This works well the only problem i have now is how to get the name of the file to download in an action?
Your help is very much ch appreciated

[kaba]

i managed to find a way to download the file to a local disk by modifying the RemoteFileMessageComposer.java class as fo lows: See the bold highlighted section

package org.jboss.soa.esb.listeners.gateway;

import org.jboss.soa.esb.listeners.message.MessageComposer;
import org.jboss.soa.esb.listeners.message.MessageDeliverException;
import org.jboss.soa.esb.listeners.ListenerUtil;
import org.jboss.soa.esb.listeners.ListenerTagNames;
import org.jboss.soa.esb.helpers.ConfigTree;
import org.jboss.soa.esb.ConfigurationException;
import org.jboss.soa.esb.addressing.EPR;
import org.jboss.soa.esb.addressing.eprs.FTPEpr;
import org.jboss.soa.esb.util.*;
import org.jboss.soa.esb.message.Message;
import org.jboss.soa.esb.message.MessagePayloadProxy;
import org.jboss.soa.esb.message.body.content.BytesBody;
import org.jboss.soa.esb.message.format.MessageFactory;
import org.jboss.internal.soa.esb.assertion.AssertArgument;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.FileChannel;
/**
* Remote File Message Composer.
* <p/>
* The logic in this class was lifeted from {@link RemoteGatewayListener}.
*
* @see org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer

*/
public class BZPARemoteFileMessageComposer<T extends File> implements MessageComposer<T> {

private FTPEpr ftpEpr;
private File downloadDir;
private MessagePayloadProxy payloadProxy;

public void setConfiguration(ConfigTree config) throws ConfigurationException {
String output_dir = config.getAttribute("output_dir");
System.out.println("Getting******************************************************"+output_dir);
EPR epr = ListenerUtil.assembleEpr(config);

if (!(epr instanceof FTPEpr)) {
throw new ConfigurationException("This Gateway only accepts FTP and SFTP.");
}
ftpEpr = (FTPEpr) epr;
// This may look a bit odd... The "file" input dir is our FTP download dir...
downloadDir = AbstractFileGateway.getFileInputDirectory(config);
payloadProxy = new MessagePayloadProxy(config,
new String[] {BytesBody.BYTES_LOCATION},
new String[] {BytesBody.BYTES_LOCATION});
}

public Message compose(T inputFile) throws MessageDeliverException {
AssertArgument.isNotNull(inputFile, "inputFile");

Message message = MessageFactory.getInstance().getMessage();
System.out.println("Setting******************************************************");
message.getProperties().setProperty("FILE_ABSOLUTE_PATH",inputFile.getAbsolutePath());
message.getProperties().setProperty("FILE_NAME",inputFile.getName());
System.out.println("Setting******************************************************"+message.getProperties().getProperty("FILE_ABSOLUTE_PATH")+ "NAME:"+message.getProperties().getProperty("FILE_NAME"));
try {
payloadProxy.setPayload(message, getPayload(inputFile));
} catch (IOException e) {
throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
} catch (RemoteFileSystemException e) {
throw new MessageDeliverException("Error reading remote input file '" + inputFile.getAbsolutePath() + "'.", e);
}

return message;
}

protected Object getPayload( T inputFile ) throws RemoteFileSystemException, IOException {
return getFileContents(inputFile);
}

public Object decompose(Message message, T inputMessage) throws MessageDeliverException {
return payloadProxy.getPayload(message);
}

private byte[] getFileContents(File file) throws IOException, RemoteFileSystemException {
RemoteFileSystem remoteFileSystem = null;
File temp = File.createTempFile("FTPdown", ".tmp");
byte[] returnBytes;
try
{
remoteFileSystem = RemoteFileSystemFactory.getRemoteFileSystem(ftpEpr, true);
remoteFileSystem.setRemoteDir(FtpClientUtil.fileToFtpString(downloadDir));
remoteFileSystem.downloadFile(file.toString(), temp.getAbsolutePath());

returnBytes=FileUtil.readFile(temp);

File target= new File("/tmp/prop/"+file.getName().substring(0,file.getName().lastIndexOf(".")));

FileChannel in = (new FileInputStream(temp)).getChannel();
FileChannel out = (new FileOutputStream(target)).getChannel();
in.transferTo(0, temp.length(), out);
in.close();
out.close();

return returnBytes;

} finally {

temp.delete();
if (null != remoteFileSystem) {
remoteFileSystem.quit();
}
}
}
}

as you can see i still have to find a way to add the output_dir to the ftp message filter or to the ftp-provider config in order not to hardcode the path "/tmp/prop/" i wonder how?
Thank you for your Help