S3Upload microservice uploads all the files to a specified bucket.

Icon

Get the Access Key ID and Secret Key from the AWS Management Console at https://console.aws.amazon.com/s3.

Icon

Provide AmazonS3FullAccess policy to the user to get the full access to the S3 service.


Configuration and Testing

Component Configuration

The figure below illustrates the Component Property Sheet  (CPS) panel with Expert Properties view enabled.


Figure 1: Component Configuration properties for S3Upload

Process Message Based on Property

The property helps components to skip certain messages from processing.

Icon

Refer the Process Message Based On a Property section under the Common Configurations topic.

Error handling configuration

The remedial actions to be taken when a particular error occurs can be configured using this attribute. 

Click the ellipsis button against this property to configure Error Handling properties for different types of Errors. By default, the options Log to error logs, Stop service and Send to error port are enabled.

Refer Error Handling section in Common Configurations for detailed information.

AWS Connection Details

Click the AWS Connection Details ellipsis  button to configure the properties.


Figure 2: AWS Connection properties

Access Key

Specify the Access Key ID.

Secret Key

Specify the Secret Key

Region

The region of AWS Management console.

Chunk Size

Chunk Size should be more than or equal to 5 MB. If the file size is more than 5MB, it is uploaded as parts of specified chunk sizes. If the provided file is less than 5MB, it will be uploaded as a single chunk.

Bucket Name

Specify the S3 bucket name. As the component does not create the bucket, specify the existing one only.

Server Side Encryption?

Select this option for the server to encrypt the data before storing in the bucket.

Server Encryption Mode

This option appears only when the Server Side Encryption? property is selected. In this case,  Amazon S3 manages the keys required for encrypting by itself.

File Name Prefix

The specified prefix is prepended to the uploaded file.

Storage Mode

The current API provides three modes of Storage classes.

  • STANDARDDefault storage class; this storage class is ideal for performance-sensitive use cases and frequently accessed data.
  • Standard Infrequent Access (Standard - IA)Amazon S3 Standard - Infrequent Access (Standard - IA) is an Amazon S3 storage class for data that is accessed less frequently, but requires rapid access when needed.
  • Reduced RedundancyThe Reduced Redundancy Storage (RRS) storage class is designed for noncritical, reproducible data stored at lower levels of redundancy than the STANDARD storage class, which reduces storage costs.

Correlation Id Property

Specify the Correlation Id message property name. The input messages will be grouped based on the correlation Id property. Messages which has the same property value will be treated as a single file. This value is appended to the FileNamePrefix property, if specified, and treated as the file name.

Completeness Condition Property Name

Name of the Completeness Condition Property. Component treats that the file was completed on receiving a message with completeness property equals to completeness property value.

Completeness Condition Property Value

Value of the Completeness Condition property. Component treats that the file was completed on receiving a message with completeness property equals to completeness property value.

Bucket Name Property

Specifies the Bucket name property name. Bucket name can be provided from the input whose value takes precedence over the one provided in CPS.

Is Message Headers a priority?

Considers Message Headers prior to the message.

If this property is enabled, the message headers may be specified in the input message as below:

Header nameDescription
SourceFileName

Name of the file to be uploaded along with its extension.

Example

Icon

ftpput1.txt

Icon

Provide the absolute path along with the file name if the SourceFilePath is not being provided.

Example

Icon

D:\FileReader\ftpput1.txt

SourceFilePath

Complete path to the File location without specifying the SourceFileName.

Example

Icon

D:\FileReader

Icon

This Header name is optional; use when the path is not provided in the SourceFileName.

FileName

Specify a name by which the file has to be uploaded.

Example

Icon

Upload1.txt

Icon

This Header name is optional; use only if the original filename (SourceFileName value) has to be changed.


Figure 3: Message Headers configured in the Feeder corresponding to the "Is Message Headers a priority?" property

Sequence Number Property

This property is used to identify duplicate sequence numbers; discards the duplicate input arrived at its input port.

Is Input Binary

Specifies whether the input messages are byte messages or not.

Is Text Base64 Encoded

Specifies if the input text needs decryption using Base64.

Threadpool Configuration

This property is used when there is a need to process messages in parallel within the component, still maintaining the sequence from the external perspective. 

Icon
  • This needs to be used only in such circumstances.
  • If sequential processing is not required, please use sessions on the input port.

Click the Threadpool Configuration ellipsis  button to configure the Threadpool Configuration properties. 


Figure 4: Threadpool Configuration properties

Enable Thread Pool

Enable this option to configure the properties that appear as below.

Pool Size

Number of requests to be processed in parallel within the component. Default value is '1'.

Batch Eviction Interval (in ms)

Time in milliseconds after which the threads are evicted in case of inactivity. New threads are created in place of evicted threads when new requests are received. Default value is '1000'.

Push all the Messages at once?

If enabled, all the currently processed files which are waiting for the completeness condition, along with current file get uploaded.

Icon

This property is not applicable if

  • the 'Is Message Headers a priority?' property is enabled
    as well as
  • any of the headers fail the validation.
Icon

It is recommended to add the following value in Runtime Arguments in the Properties pane, which represents the maximum memory allocated to process files from the Harddisk instead of RAM (this memory will be automatically freed once the process finishes).


Figure 5: Configuring the runtime arguments to manage memory

Functional Demonstration

Uploading a file into the AWT Management console and displaying the status. Configure S3Uploader as described in the Configuration and testing section and use Feeder and Display components to send sample input and check the response respectively.


Figure 6: Demonstrating a scenario with sample input and output

Input Message


Figure 7: Input XML sent using feeder

Output Message


Figure 8: Output demonstrating the status of the uploading file

The AWS Console displays the detailed state of storage class (Reduced Redundancy in the case of test34.pdf).


Figure 9: AWS Console details

Encryption status used to store the data.


Figure 10: Encryption status

Adaptavist ThemeBuilder EngineAtlassian Confluence