Objective

To demonstrate the capability of FTPGet component to monitor files in a Source location on SFTP server using Private Key authentication and Highest Modification Time filter condition.

Prerequisites

• Start Fiorano Enterprise Server (FES) and Fiorano Peer Server (FPS)
• Start SFTP server. If SFTP server is not installed, refer the Installing SFTP Server section below.
• Login to eStudio application
• Understand the basic menus and perspectives (panels) in eStudio application

Scenario

Using FTPGet Component, download files whose last modification time is greater than the highest modification time found in the last polling cycle to local destination from SSH protected FTP server.

Components used

• FTPGet
• Display

Setting up the Example

Installing SFTP Server

Install the SFTP server in Linux following the steps explained below (please ignore if the installation is already done):

1. Install vsftpd server by executing the following command in the Command prompt:
   yum install vsftpd
2. Generate keys pairs (both RSA and DSA) and store in ~/.ssh/ directory. For key generation we can use the following commands for RSA type and DSA type key generations respectively:
   ssh-keygen -t rsa
ssh-keygen -t dsa
3. While executing the above commands, file name and password for the file has to be provided. If the file name provided is id_rsa, then two files will be generated named id_rsa.pub and id_rsa which are public key and private key respectively.
4. Install the public keys in server. Private key has to be with the client which is used to connect to the server. The SFTP server will authenticate the client's private key using its public key. This type of client-authentication is called Private Key Authentication. The installation of the public key in the server can be done by executing the following command:
   ssh-copy-id -i ~/.ssh/id_rsa.pub root@localhost
   where "~/.ssh/id_rsa.pub" is the rsa public key file path and "root@localhost" is the server (here we are using the same machine).
5. Change the following in sshd_config file located at etc/ssh/ folder:
   1. Set RSAAuthentication or DSAAuthentication to 'yes' based on the key file type used.
   2. Add the line "IdentityFile ~/.ssh/id_rsa" in the ssh_config file located at /etc/ssh (if already Identity File is set to some other file, modify it).
6. Restart the servers using the following commands:
   /etc/rc.d/init.d/sshd restart
/etc/rc.d/init.d/vsftpd restart

After performing the above actions, the server will be ready to accept SFTP protocol from clients to connect to it. Server settings in FTP component help to connect to the server and perform Put/Get operations on the server.

Configuring FTPGet component

FTPGet component helps to download file from the FTP Server. To configure FTPGet component, perform the following actions:

1. Add a new Event Process (EP) with name 'SFTP_Monitoring_Eg'.
2. Drag the FTPGet component from the Bridges category in the Micro Service Palette to the Fiorano Orchestrator.
   
   
   Figure 1: Adding FTPGet component in SFTP_Monitoring_eg EP
    
3. Double-click the FTPGet component to open its Configuration Property Sheet (CPS).
   
   
   Figure 2: FTPGet CPS

SFTP Protocol Configuration

To enable Secured File Transfer, SFTP protocol needs to be set in connection configuration. To configure STP protocol, perform the following actions:

1. In theManaged Connection Factory window, click Connection Properties tab to expand the options and click the Connection Configuration Ellipsis button.
   
   
   Figure 3: Navigating to do Connection Configuration
    
2. In FtpConnectionConfig dialog box, select 'SFTP' from the Protocol drop-down under Component Configuration section and provide the following values menu to load the SFTP server configuration settings:
   1. Remote host: The IP address of the machine where the SFTP server is running. In this example, it is 'localhost' as the server is installed in the same machine.
   2. Port: 22, the port number on which the FTP server is running.
   3. Login: Username to be used to login to the SFTP server . In the example, it is root, the user under which the server had been installed in the machine.
   4. Client Authentication Type: Select 'Private Key' from the drop-down to login to the SFTP server using the client's private key. Three fields needed for private key authentication appear: Private Key Type, Private Key file and Private Key Password.
   5. Provide the path of the private key (id_rsa) generated above in the Private key field. Select the Private key type as RSA/DSA depending on the generated key. Enter the Private key password also to provide access to the key by the server.
      
      
      Figure 4: SFTP protocol configuration
       
3. Click OK to save the server configuration.
4. After saving the connection configuration details, click the Test button in Managed Connection Factory page. Test Connection Configuration pop-up shows the connection status.
   
   
   Figure 5: Testing SFTP Protocol connection

Monitor Directory Configuration

To make the FTPGet adapter poll a directory on the server for files matching a particular pattern and download all such files from the server. Enabling this property makes the FTPGet adapter poll the Source directory using the polling configuration specified in Scheduler Configurations panel. After ensuring that the Source directory exists on server, perform the following actions:

1. In the Managed Connection Factory page, click Next to move to the Interaction Configurations Page.
   
   
   Figure 6: Interaction Configurations panel in FTPGet CPS
    
2. Select the Monitor Directory checkbox to enable Monitoring on the SFTP server. Three options appear as below:
   1. Monitor Directory Configuration
   2. Monitor Time Configuration
   3. Use File not FoundException in Monitoring
   
   Figure 7: Selecting Monitor Directory option
    
3. Click the Monitor Directory Configuration Ellipsis button to open the FtpMonitoringConfiguration window.
   
   
   Figure 8: Navigating to Monitor Directory Configuration
    
4. In the FtpMonitoringConfiguration window, provide the following details:
   
   1. Enter the folder path on the server where the files need to be monitored. Enter paths for Working directory,Processed directory and Error directory also and all of these paths should be different from each other.
   2. Select Move to working directory checkbox, if the files which are processed need to be moved to Processed directory after the transfer is complete. Also, files will move to error directory in case of any unexpected error during the transfer. If the checkbox is cleared, then files will remain in the source folder even after the processing is complete.
   3. Provide value for File name patterns to filter specific file types that needs to be monitored in the source folder. In this example, it is "*.csv", which picks all the .csv files in the source folder for processing.
   
   Figure 9: Configuring Directory details
    
5. Click OK to save the Monitoring Directory configuration.

Monitor Time Configuration

Different Time filters help to filter the files while monitoring the source folder based on their last modified time.

To configure the component with Highest Modification Type filter type as in the example scenario, perform the following actions:

1. Under Monitoring Settings section in the Interaction Configurations panel, click the Monitor Time Configuration Ellipsis button.
   
   
   Figure 10: Navigating to Monitor Time Configuration
    
2. In the FtpMonitorTimeConfiguration dialog box, select Time-based filefiltering type as 'HIGHEST_MODIFICATION_TIME' and then click OK.
   To understand the functionality of each Time filter type, refer <link to FTPGet Help>
   
   
   Figure 11: Selecting HIGHEST_MODIFICATION_TIME filter type
    
3. For HIGHEST_MODIFICATION_TIME filter type, two other options namely Base Time and Remote host time offset appear.
   
   
   Figure 12: Configuring Highest Modification Time filter type details
   
   
   1. Base Time: 23:11:2012 15:25 During monitoring, the Base Time value will be used only in the first poll for this filter type. Only the files with last modified time greater than the base time will be picked in the first poll from the source folder.
   2. Remote host time offset: -03:30This will be the time difference between the SFTP server time and client machine local time.

 

 

Target Directory Configuration

In the Interaction Configurations panel, perform the following actions:

1. In the Parent directory path on the local system text field under Target Settings section, enter the Parent Directory path relative to which a relative path of Target directory would be computed. Provide it as D:\dev\ftp
   
   
   Figure 13: Providing parent directory
    
2. Click the Target Directory Configuration Ellipsis button to configure target directory.
3. In the TargetConfig dialog box, make the following changes and click OK to save the Target Directory Configuration:
   1. Target Directory: Enter the path relative to the parent directory provided in the previous step. Files which are processed while monitoring the source folder will be saved to this folder after the transfer gets completed. Provide it as 'receivedFiles'.
   2. Select Use Temporary target directory checkbox to ensure that only completely transferred files will be saved in the target directory. Temporary target directory will be named as 'temp' by default.
   3. Select appropriate option (Overwrite/Rename/Append) from the Action if same file exists drop-down to define the action when the file being transferred matches with the file name of an already existing file. Select OVERWRITE.
      
      
      Figure 14: Providing Target Directory details
       
4. Under Target Settings section, select Auto create Target Directory checkbox to automatically create target directory at the given path if it is not already present on the system.

Monitor Scheduler Configuration

To configure scheduling information, continue FTPGet configuration as follows:

1. In the Interaction Configurations panel, click Next to navigate to the Scheduler Configurations panel.
   
   
   Figure 15: Providing Scheduler Configurations
    
2. Under Scheduler Configurations tab, set Repeat Interval text field as 5 setting the time gap required between each poll while monitoring the source folder for any matching files.
   To set how many times the poll should run on the folder before stopping the poll, clear the Repeat forever checkbox and provide a value in the Repeat Only 'n' Times field. In this case, retain it as it is.
   Advanced scheduling options can be found in the Advanced Configuration tab. Daily/Weekly/Monthly polls with specific Start Time and End Time Sessions is possible.
3. Click Finish to save the Component configuration.

After the above steps are completed, the FTPGet component gets configured for monitoring the files in the source folder on the SFTP server.

Composing the Event Process

To add a Display component to display the output response from the FTPGet component after each file transfer gets completed in the SFTP_Monitoring_Eg event process, perform the following actions:

1. Drag a Display component also to easel from Micro Service Palette under Util category.
2. Connect the output port of FTPGet component to input port of Display component.
   
   
   Figure 16: Connecting FTPGet component to Display

Running the Example

Ensure that the SFTP Server is running and perform the following actions:

1. Click Check Resource and Connectivity icon (or press ALT+SHIFT+C) to check resource and connectivity.
2. Click Run Event Process icon (ALT+SHIFT+R) to run the event process. Display window gets opened.
3. Add a file 'Newfile.csv' with last modified time as '12 March 2014 08:09:25 PM' in the source folder after the second poll.
4. Display window gets the message when the file transfer happens.
   
   
   Figure 17: Message appearing in Display after the File Transfer

The manner in which the application works is explained in the below section.

How it Works

For SFTP Monitoring with Highest-Modification Time-filter type enabled, the event process goes through the following steps during the processing:

1. Source folder configured in FTP component contains the following files with the corresponding time of modification:
   1. CheckHead_01232014.csv: Tue 11 Mar 2014 11:00:20 PM IST
   2. CheckItems_01232014.csv: Wed 12 Mar 2014 07:52:55 PM IST
   3. Data.txt: Wed 12 Mar 2014 07:39:16 PM IST
   4. Hello.txt: Thu 23 Jan 2014 05:48:34 PM IST
   5. Number.txt: Wed 12 Mar 2014 07:38:57 PM IST
   
   Figure 18: Contents in the Source Folder
2. In the first poll, after launching the event process (12 March 2014 08:02:15 PM in this example), while monitoring the folder, only the files 'CheckHead_01232014.csv' and 'CheckItems_01232014.csv' will be picked for processing, as they match both the pattern name condition (.csv file type) as well as the HIGHEST MODIFICATION TIME filter condition for first poll. Both the files have last modification time greater than the Base time provided in the Monitor Time Configuration.
3. After the first poll gets completed and all the matching file transfers are finished, the component calculates the Highest Modification Time among all the processed files and save it for next poll. In this case, it will be '12 March 2014 07:52:55 PM'.
4. In the Second poll (at 12 March 2014 08:07:15 PM as per Repeat Interval - 5 minutes), none of the files will be processed, as all the files have last modification time less than or equal to the Highest Modification Time calculated in the previous poll.
5. After Second poll, add a file 'new.csv' with last modified time as '12 March 2014 08:09:25 PM' in the source folder after the second poll. (please do this manually)
6. In the Third poll, the Highest Modification Time value will still be '12 March 2014 07:52:55 PM' as it did not process any new files in the previous poll. Now the file named Newfile.csv will be processed in the third poll (at 08:12:15 PM) as its last modification time is greater than the Highest Modification Time value and matches the file pattern name. The FTPGet Output response appears in Display window (as provided in "Running the Example" section) after the transfer gets completed.
7. After the transfer of Newfile.csv file gets completed, the Highest Modification Time value will be set to the last modified time of the Newfile.csv file as it is the only file processed during the third poll and the component will continue its polling as per the Scheduler Configuration.

Reference

• Please note that the component options used in this example are minimum, which helps you to get an overview of the application. To explore the other options present in the FTPGet and Display components, refer: FTPGet and Display sections.