The FTPPut component is used for uploading files to the FTP Server. It can be used for uploading one or more files in a directory.

Using the FTP Put component, you can upload the files in any one of the following methods:

• By monitoring file(s) in a local directory for any additions or modifications and uploading the corresponding files to the desired location on the remote host.
• By uploading file(s) specified in the input message.
• By the data contained in the input message.

The FTP Put component uses the FTP protocol for file transmission. The component ensures uninterrupted upload of file by attempting to reconnect to the remote server in case the connection to the server is lost.

Points to note

• It is not mandatory to provide the remote file name. If the remote file name is not provided or it is null, then the remote file name is extracted from the Local File Path. Providing local file path is mandatory.
• The remote file path is relative to the ftp server.
• The component runs on the peer server and therefore the file paths and directories mentioned in the CPS should be valid on the machine where the peer server is running. If the component fails over to another peer, ensure that the machine on which the secondary peer server is running does have the same path available.

Configuration and Testing

Managed Connection Factory

The connection properties can be configured using the properties of Managed Connection Factory panel as shown in Figure 1.

Figure 1: Managed Connection Factory panel

Connection Properties

Use Connection Details From Input

Parameters to create the connection can be specified in the input message when this property is enabled. If this property is selected the validation errors in the managed connection factory panel of the CPS are treated as warnings. So user can bypass this step without giving valid configuration and complete the configuration of the component. If valid properties are not provided even in the input message exception will be thrown at runtime.

Connection Configuration

Protocol

• FTP - File Transfer Protocol is used for transferring files to FTP server.
• SFTP - Secure File Transfer Protocol is used for file transfers. If selected, then the property Client Authentication Type is enabled. If Client Authentication Type is set to Password, then client's user name and password are sufficient to login successfully. These details can be set by using the properties Login and Password. If Client Authentication Type is set as Public Key or Both, then the details of Private Key File, Key File Type and Key File Password has to be provided. For detail explanation of SFTP setting, please refer Scenario 3 under section Functional Demonstrations.
• FTPS- FTPS is an extension to the commonly used File Transfer protocol(FTP) that adds support for the Transport Layer Security(TLS) and the Secure Sockets Layer(SSL) cryptographic protocols. FTPS should not be confused with the SFTP protocol,an incompatible secure file transfer subsystem for the Secure Shell(SSH) protocol . It is also different from the Secure FTP, the practice of tunneling FTP through an SSH connection.
  FTPS option enables the ImplicitFPS feature. Implicit FTPS takes SSL one step further than simply requiring that SSL- related commands must be sent first like you can with Explicit SSL; with Implicit FTPS, an SSL handshake must be negotiated before any FTP commands can be sent by the client. In addition, even though Explicit FTPS allows the client to arbitrarily decide whether to use SSL, Implicit FTPS requires that the entire FTP session must be encrypted. Basically the way that Implicit FTPS works is that an FTP client connects to the command/control channel, in this case using port 990 instead of using port 21 as in the case of FTPS, and immediately performs an SSL handshake; after SSL has been negotiated, additional FTP commands for the session can be sent by the FTP client. A Client Authentication type needs to be specified even here.

FTP Protocol Settings

Figure 2: FTP enabled Connection Configuration

Remote Host

The host name/IP address of the machine where the FTP server is running.

Port

The port number on which the FTP server is running.

Login

User name of the FTP user.

Password

Password of the FTP user. This field will be disabled if user selects Protocol as SFTP and Client Authentication Type as private Key.

SFTP Protocol Settings

All properties except the below ones remain the same.

Timeout (in ms)

The TCP timeout in milliseconds for the sockets. Any operation which takes longer than the timeout value is killed with a java.io.InterruptedException.

Client Authentication Type

Figure 3: Client Authentication Type - SFTP

This property determines the authentication type for the client validation for the specific protocol.

• Password: If selected, client's username and password are used for client's authentication.
• Private Key: If selected, client's Private Key and the Key Password are used for client's authentication. (Key file password is different from client password). This option is available only in SFTP protocol.
• Both: This option authenticates using a private/public key-pair, followed by password authentication. If the authentication fails while using client's private key, then it will try to authenticate using password authentication. This option is available only in SFTP protocol.

Private Key file

The private key file path in the local machine used for client authentication in case of protocol SFTP. The path should include the file name also. The key file should be present on the machine where the peer server (on which peer the component is running) is running. This property is visible when the Protocol is selected as SFTP and Client Authentication Type as Private Key or Both.

Private Key Type

Determines the private key type either 'DSA' or 'RSA'. This property is visible when the Protocol is selected as SFTP and Client Authentication Type as Private Key or Both.

Private Key password

The private key file's password when the property protocol is set to SFTP and the property Client Authentication Type is set to Private Key or Both.

FTPS Protocol Settings

All properties except the below ones remain the same.

Implicit FPS

To enable implicit FTPS ,when using the FTPS protocol.

Client Authentication Type

This property determines the authentication type for the client validation for the specific protocol.

Figure 4: Client Authentication Type options for FTPS

• Password: If selected, client's username and password are used for client's authentication.
• Password and KeyStore: This option authenticates using both password and KeyStore. The keystore file of JKS type is used to load Private key and Certificate required to authenticate the client apart from password authentication. This option is visible only for FTPS protocol.

Figure 5: Properties supporting Authentication Type 'Password and KeyStore'

 

KeyStore File (JKS)

The KeyStore file path in the local machine used for client authentication in case of protocol FTPS. The path should include the file name also. It should be of JKS type. The KeyStore file should be present on the machine where the peer server (on which peer the component is running) is running. This property is visible when the Protocol is selected as FTPS and Client Authentication Type as Password & KeyStore. It is used to load Private key and Certificate required to authenticate the client apart from password authentication.

KeyStore Password

The Password for the KeyStore File given above. This property is visible when the Protocol is selected as FTPS and Client Authentication Type as Password & KeyStore.

KeyStore Alias

The Alias name used for the KeyStore File given above in th Keystore file path. This property is visible when the Protocol is selected as FTPS and Client Authentication Type as Password & KeyStore.

Private Key Password

The private key file's password when the property protocol is set to FTPS and the property Client Authentication Type is set to Password & KeyStore.

Proxy Settings

For Proxy Settings, Connection Pool Params and SSL Security, refer the corresponding sections in FTPGet page.

Advanced Settings

Figure 6: Advanced Settings

Connect mode

Determines the type of FTP connection – Active or Passive. This property is ignored if Protocol is set as SFTP.

Figure 7: Connect Mode

• Active: In Active mode the FTP client specifies the data port that the FTP server is going to connect on and waits for the FTP server to connect. The IP address and port numbers are sent to the FTP server by the FTP client using the PORT command.
• Passive: In passive mode the FTP server specifies the data port on which the FTP client connects and waits for the FTP client to connect. The FTP client will ask the FTP server for the server's IP address and port number by issuing the PASV command to the FTP server. This will usually solve the problem of firewalls filtering the incoming data connection.

Transfer type

Specifies the data transfer type

Figure 8: Transfer Type

• Ascii: The transferred data is considered to contain only ASCII formatted text. The component is responsible for translating the format of the received text to one that is compatible with the operating system of fps (The fps on which the FTPGet component is running). Text files and files containing HTML, CSS mark-up are suitable for Ascii mode transfer.
• Binary: The component transmits raw bytes of the file being transferred. All audio, video and image files are suitable for Binary mode transfer.

Resume Transfer

Resumes FTP transfer from the point where download has stopped if the transfer is broken. Resume of the broken transfer depends on the FTP server. If the FTP server does not support this then the FTP adapter will start from the beginning otherwise it will start from where it was stopped.

Extensions of the files to be filtered

If files with specific extensions have to be restricted for download from the server, the file extension has to be specified here. This property accepts comma separated list of file extensions. Example: *.zip, *.exe, *.dat.

Example: If this property is set to .exe and the user specifies to put the file named "installer.exe" in the request, then the component ignores that request.

Debug responses

When FTP responses are needed, enabling this property logs all the FTP responses to the Output Log of the component. Below is a sample snapshot of the debug responses during a download.

Figure 9: FTP responses in FTPPut's output log

Advanced Details

Figure 10: Advanced Configuration dialog box

Current directory

The directory on FTP server to which the user's current working directory will be changed after the login to FTP. The behavior is similar to executing the command cd <directory_name> after logging in, where <directory_name> is the value provided for this property. All relative paths in the server that are computed by the FTPPut component are relative to this directory.

Example: If the default working directory for the user is /home/user and current directory is set to /home/user/Fiorano, then the working for the user will be changed to /home/user/Fiorano after the user logs into the FTP server.
If the value of property Use Temporary target directory is enabled and the value for property Temporary target directory is set to temp, then a directory temp will be created under the directory specified by this property.

SITE command parameters

Site commands are sets of extended commands that can be issued by a FTP client, and they are not defined in RFC. However, they are supported by different FTP servers, and different servers usually have different supported site commands. SITE command is used by the server to provide services specific to the system. All the server administrative tasks can be performed by the SITE command.

This property accepts a semicolon separated list of SITE command parameters that have to be executed immediately after login. These parameters are server dependent.

Example: For OS/400 platform, the server specific format of lists or names can be changed to Unix type formats by specifying the value LISTFMT 1; NAMEFMT 1 for this property.

Use specified format for parsing Directory Listing?

This property is used to parse the directory listing of the FTP Server. For example, in case of Unix, the directory is listed as follows:drwxrwxr-x 3 user group 4096 2008-10-23 14:13 fioranodrwxr-xr-x 14 user group 4096 2008-12-18 14:41 Fiorano
But on Windows, the directory is listed in different format. This listing is the output from the FTP server after executing the dir command.

• When this property is disabled, the parsing format will be chosen depending on the operating system on which the FTP Server is running.
• Enable this property if a specific parsing format is to be used for parsing the Directory listing returned by the FTP Server.

Example: If a FTP Server is running on IIS on a Windows machine and its directory listing style is set to Unix, enable this property and set 'Parsing format for Directory Listing' to Unix for this property.

Parsing format for Directory Listing

This property is used to determine the format to use for parsing directory listing in the FTP Server. The formats supported by the component are Windows and Unix.

Figure 11: Parsing format for Directory Listing

• Unix: Select to use Unix format to parse the directory listing
• Windows: Select to use Windows format to parse the directory listing

Testing the Connection

Server connection can be tested from within the CPS by clicking on the Test button in the Managed Connection Factory panel.

Figure 12: Result of successful connection creation

After selecting the appropriate parameters, click the Next button to navigate to Interaction Configurations panel.

Interaction Configurations

Figure 13: Interaction Configurations

Source Settings

Response type

Specifies the type of user input to the adapter. This property provides two options:

• File
• Data

When the request type is File, the path of the local file which is to be transferred is specified in the input.

Refer to section Input and Output for details about the effects of these configurations on input and output structures.

Send Transfer Progress Notifications

When large files are being uploaded to the FTP server, the progress of the transfer can be obtained by specifying yes to this property. If this property is selected as Yes then the FTP adapter will send the notifications of the uploaded process at regular time intervals and this time interval can be specified by the property Monitor Progress interval (in ms). The example output of this notification is shown below. Now, observe the BytesTransferred and TotalBytes fields in the output XML.

Figure 14: Sample output when 'Send Transfer Progress Notification' is set.

Other Source Settings Properties

• Validate Input: If this property is enabled, FTPGet adapter will validate the input request with the input port xsd.
• Monitor Progress interval (in ms): The time interval (in milliseconds) between any two progress notifications. This is visible when the Send Transfer Progress Notifications property is enabled.
• Cleanup resources (excluding connection) after each document: This closes all the resources except to connection used by the FTPPut adapter after every request. If the less processing time is more important than the less memory usage, then it is recommended to disable this property and vice versa.
• Delete file after transfer: Specifies if the remote file is to be deleted after it is completely uploaded.
• Target Namespace: Target Namespace for the FTP request and response XML messages.

Monitor Settings

Monitor Directory

This property can be used to make the FTPPut adapter poll a directory on the peer server's system(on which the component is launched) for files matching a particular pattern and upload all such files to the server. Enabling this property makes the FTPPut adapter poll the Source directory using the polling configuration specified in Scheduler Configurations panel. The user has to make sure that the Source directory exists.

Monitoring is done by monitoring the source directory in regular scheduling interval. The scheduler configurations can be defined by the user in Scheduler Configurations Panel and this is the only case that FTPPut component uses scheduler configurations.

Figure 15: Monitoring Settings properties

Parent directory on the local system

Path of the directory relative to which the paths of Working, Processed and Error directories should be computed

Monitor Directory Configuration

Figure 16: Sample monitoring configuration

Source directory

The directory which contains the files to be uploaded. FTPPut component polls this directory using the polling settings configured in Scheduler Configurations panel.

File name patterns

The type of files in the Source directory which are to be picked up and downloaded. This property accepts multiple file name patterns separated by pipes. Example: .txt|.xml|*.exe

Move to working directory

When this property is enabled, the file that has to be uploaded to FTP server will be moved from the directory specified by property Source directory to the directory specified by the property Working directory before the upload begins. The file is read from the working directory and uploaded to the FTP server.

• If the upload is successful, the file is moved from working directory to the directory specified by property Processed directory.
• If the upload is not successful, the file is moved from working directory to the directory specified by property Error directory.
• When this property is disabled, the file is read directly from the directory specified by property Source directory.

 

 

Working directory

This directory holds the files for which the file transfer is in progress.

Processed directory

This directory holds the files for which the upload has been successful.

Error directory

This directory holds the files for which the upload has failed.

Monitor Time Configuration

Time-based file filtering type

This property provides the capability of monitoring only specific files depending on their modification times. This property provides 5 options based on which the files to be monitored could be filtered.

Figure 17: Time-based file filtering types

The behavior of each filtering option is explained below.

NONE

No filtering is applied on the files. Every file present in the Source directory is monitored.

TIME

Files whose last modification time is greater than the last polling time is monitored. This ensures that only the files modified/added after the last polling cycle are monitored.


Figure 18: Time-based file filtering type - TIME

• Base Time: Base time in dd:MM:yyyy hh:mm format after which the changed files are to be uploaded

 

HIGHEST MODIFICATION TIME

Files whose last modification time is greater than the highest last modification time found in the last polling cycle is monitored. This ensures that only files which are newer than the newer file already polled are selected.


Figure 19: HIGHEST MODIFICATION TIME Filter Type

 

MINIMUM AGE

Files whose last modification time is less than the current polling time minus the age is monitored. This ensures that the file modification time is at least Minimum Age earlier than the current time.


Figure 20: MINIMUM AGE filter type

• Minimum age: The minimum age of the files which are to be monitored.

MAXIMUM AGE

Files whose time difference between the current scheduling time and the last modification time is less than the specified max age will be monitored. This ensures that the file modification time is not older than MaximumAge time for the current poll.


Figure 21: MAXIMUM AGE filter type

• Maximum age: The maximum age of the files which are to be monitored.

 

LAST RECENT POLL TIME

Files whose last modification time is greater than the last recent poll time will be monitored. This ensures that only files which are newer than the last recent poll time in the system are selected.

 

 

Figure 22:  LAST RECENT POLL TIME filter type

 

 

Example
Let us say if the source directory contains 4 files named a.txt, b.txt, c.txt and d.txt. The scheduling interval is 3 minutes and the first poll is going to start at 11:00:00 (This scheduling settings can be configured in Scheduler Configuration Panel, please refer the section "Scheduler Configuration Panel" for more details)

At first poll all files will be monitored irrespective of the value of the property "Time-based file filtering type". 

Consider that the files have last modification time as below:

 

• a.txt 11:00:16
• b.txt 11:00:30
• c.txt 11:02:10
• d.txt 11:02:50

Following are the various outcomes for different filtering types. If the property Time-based file filtering type is set to:

• TIME: All the files will be monitored in the next poll (which is going to poll at 11:03:00), since all files are modified after the last poll.
• HIGHEST_MODIFICATION_TIME, then also all files will be monitored in the next poll (which is going to poll at 11:03:00), since all files are having the last modification time greater than the highest last modification time found on last poll (Component will keep the track of highest last modification time found in the poll).
• MINIMUM_AGE' with Minimum age set to 5 min, then no files will be monitored in the next poll (which is going to poll at 11:03:00). Files a.txt and b.txt will be monitored in the scheduling which will be going to poll at 11:06:00, and the files c.txt and d.txt will be monitored in the scheduling which will be going to poll at 11:09:00, because the files monitored in the particular scheduling have been modified at least 5 min ago from the scheduling time.
• MAXIMUM_AGE: If Maximum age is set to 2 minutes or 120000 milli seconds, then the files c.txt and d.txt only will be monitored at 11:03:00, because their last modified time is less than 2 minutes from the current system time.
• LAST_RECENT_POLL_TIME: All the files will be monitored as all the files have modification time greater than the last recent poll time which is 11:00:00.

Use File not Found Exception in Monitoring

If this property is enabled, it throws a 'file not found' exception if no files matching the specified pattern and time filters are found in the monitored directory.

Target Settings

Target Directory Configuration

Click the ellipsis button the configure Target Directory.

Figure 23: Target Directory Configuration

Target directory

Directory on the FTP server to which the file(s) is/are to be transferred. Note that this property allows relative paths which would be computed relative to the directory specified for Parent Directory on the local system.

Use Temporary target Directory

If this property is enabled, then the FTPPut adapter will use a temporary target directory for intermediate processing. If you do not prefer to create an extra directory in the FTP server, disable this property.

 

Temporary target directory

This property is visible only when the property Use temporary target Directory is enabled. This is the directory on the ftp server, which the FTPPut component uses for intermediate processing during file downloads.

 

Action if same file exists

Action that must be taken if the target directory already has a file with name same as the file that is to be uploaded. The behavior will be dependent on the selection as shown below:

OVERWRITE

The file to be uploaded overwrites the one in the target directory.

RENAME

The name of the file that is being uploaded is changed based on the properties given below:


Figure 24: Appender counter

• Append date-time format postfix: When existing files in the Target directory are not to be overwritten, FTPPut provides the flexibility of uploading the content into a new file whose name is in the format <NameOfExistingFile_CurrentDateTime>. The format in which the date and time is to be appended should be specified as a value for this property.
  Example: If the date-time format is specified as MMddyyyyHHmmssss for the file Sample.txt, the target file created would be Sample_0305200811300013.txt.

• Append counter: Enabling this property appends a counter along with the Date and Time to target file name, when the target file is not to be overwritten.
  Example: A sample file name could be Sample_0305200811300013_0.txt.

 

APPEND

The file being uploaded gets appended to the one in the target directory.

 

Auto Create Target Directory

Enable this property to create target directory. If this is not set, the directory has to be created explicitly by the user.

Miscellaneous Settings

Auto create temporary directories

The temporary directories created for processing of the component are created based on this property. If this is not enabled then the temporary directories must be created explicitly.

Input and Output

The input and output structures depend the configuration of property Request type.

When Request type is set to File, input and output structures are defined as shown in Figure 25 and Figure 26 respectively.

Figure 25: Input schema structure for the request type - File

Table 1: Input schema element descriptions for - File request type

Figure 26: Output schema structure for the request type - File

Table 2: Output schema element descriptions for - File request type

When the type of input is Data, data to be transferred is provided under the Data element in the input message and the data will be written to a file with the name specified under the element RemoteFile.

If text data has to be transferred, then the text content should provided under the Data element in the input message and the attribute dataType should be set to Text.
If binary data has to be transferred, then the base64 encoded string created from binary data should be provided under the Data element in the input message and the attribute dataType should be set to Text.

Figure 27 and 28 show the input and output schema structures respectively, when the request type is Data. The only difference in these schema structures against the ones for File request type is the replacement of the schema element LocalPath with Data. Please refer Table 1 and 2 for the remaining schema elements.

 

Figure 27: Input schema structure for the request type - Data

Figure 28: Output schema structure for the request type – Data

When the property Use Connection details from input is chosen, an additional element ConnectionFactorySettings is added to the input schema, as shown in the figure. Properties that are used to create the connection are present under this element.

Figure 29: Input schema with ConnectionFactorySettings

Testing the Interaction Configurations

The configuration can be tested by sending a test file when you click on the Test option in the interaction properties panel.

Figure 30: Sample input sent from CPS

The FTP server can be configured in the connection properties panel of CPS.

Figure 31: Sample output

Functional Demonstration

Scenario 1

Send files from a local directory to the FTP server's remote directory.
Configure the FTP Put as described in Configuration and Testing section and use feeder and display components to send sample input and check the response respectively. In the interaction configuration choose the option File for property Request type.

Figure 32: Event Process demonstrating Scenario 1

Sample Input

Figure 33: Input sample when Request type is File.

Sample Output

Figure 34: Output message received for the input

Scenario 2

Send data to the FTP Server and save it as a file in the remote directory.
Configure the FTP Put as described in Configuration and Testing section and use feeder and display component to send sample input and check the response respectively. In the interaction configuration choose the option Data for the property Request type.

Sample Input

Figure 35: Input sample when Request type is Data

Sample Output

Figure 36: Output message received for input

Scenario 3 (Scenario 1 Using SFTP Protocol)

The following steps give a brief description about server settings and SFTP protocol. Here, you provide steps to test FTPPut adapter using vsftpd server which is installed in Linux.
Steps to produce:

1. Install vsftpd server by executing the following command on command prompt:

   yum install vsftpd

2. Generate keys pairs (both RSA and DSA) and store in ~/.ssh/ directory. For key generation, you can use the following commands:

   ssh-keygen -t rsa  (for RSA type key generation)  ssh-keygen -t dsa (for DSA type key generation)

   Icon

   While executing the above commands, file name and password for the file has to be provided. If the file name provided is id_rsa, then two files will be generated named id_rsa.pub and id_rsa which are public key and private key respectively

3. Next, install the public keys in server. Private key has to be with the client which is used to login. The server will authenticate the private key using its public key. This type of client-authentication is called Public Key Authentication. The installation of the public key in the server can be done by the executing following command:

   ssh-copy-id -i ~/.ssh/id_rsa.pub root@localhost ~/.ssh/id_rsa.pub is rsa public key file path and root@localhost is the server (here we are using the same machine).

4. Now, change the following in sshd_config file (this file can be located in etc/ssh/ folder). Set RSAAuthentication or DSAAuthentication based on the key file type used. 
5. Add the following line IdentityFile ~/.ssh/id_rsa in the ssh_config file which was located in /etc/ssh. (if already Identity File is set to some other file, then it has to be modified)

6. Now, restart the servers using the following command:

   /etc/rc.d/init.d/sshd restart /etc/rc.d/init.d/vsftpd restart

7. Now the server is ready to accept SFTP protocol to login.

Configuring FTPPut Component

Connection Configuration of FTPPut component for SFTP protocol is as shown in the figure below, and the remaining procedures are same as explained in Scenario 1.

Figure 37: Protocol type selection

Use Case Scenario

In the revenue control packet example error messages are sent to a FTP server and are stored there for tracking using the FTP Put component.

Figure 38: Demonstration scenario

The event process demonstrating this scenario is bundled with the installer. The bundled process shows it as a File Writer component instead of a FTP Put component.
Documentation of the scenario and instructions to run the flow can be found in the Help tab of flow when open in Studio.

Useful Tips

• Some FTP Servers have an idle timeout limit for the client connections configured at server side. When a client connection is idle beyond this time limit, server closes the connection. So a new connection has to be made, if a request is made after being idle for time greater than this idle timeout. This means, if the server has a idle timeout value set to 120s and the component does not receive a message at any point of time for more than 120s, the connection would be closed.
  
  There are multiple solutions to this problem -

• • If the component has to wait for more than 120 seconds before a new message is delivered to the component, then disabling connection pool is a better option.
  • If the component has to wait for more than 120 seconds in very few cases before a new message is delivered to the component, then enabling reconnection is a better option.
    Increase the timeout at server side to a higher value, if possible.

• If error with error code 550 occurs during request execution, the component logs the message as resource warning but not as error. To get these messages as errors, the property "Throw fault on warnings" must be selected for request processing error in error configuration panel