Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Bolo.

Table of Contents

Search

  1. Introducing Mass Ingestion
  2. Getting Started with Mass Ingestion
  3. Connectors and Connections
  4. Mass Ingestion Applications
  5. Mass Ingestion Databases
  6. Mass Ingestion Files
  7. Mass Ingestion Streaming
  8. Monitoring Mass Ingestion Jobs
  9. Asset Management
  10. Troubleshooting

Mass Ingestion

Mass Ingestion

Amazon S3 V2 target properties

Amazon S3 V2 target properties

When you define a file ingestion task with an Amazon S3 V2 target, you must enter target options on the
Target
tab of the task wizard.
The following table describes the target options:
Option
Description
Folder Path
Amazon S3 folder path to where files are transferred, including bucket name. The default value is the folder path specified in the connection.
You can enter a relative path. To enter a relative path, start the path with a period, followed by a slash (./). The path is relative to the target directory specified in the connection.
Add Parameters
Create an expression to add it as a
Folder Path
parameter. For more information, see Add Parameters.
File Compression*
Determines whether or not files are compressed before they are transferred to the target directory. Select one of the following options:
  • None
    . Files are not compressed.
  • GZIP
    . Files are compressed using GZIP compression.
File Encryption Type*
Type of Amazon S3 file encryption to use during file transfer. Select one of the following options:
  • None
    . Files are not encrypted during transfer.
  • S3 server-side encryption
    . Amazon S3 encrypts the file using AWS-managed encryption keys.
  • S3 client-side encryption
    . Ensure that unrestricted policies are implemented for the AgentJVM, and that the master symmetric key for the connection is set.
If File Exists*
Determines what to do with a file if a file with the same name already exists in the target directory. Select one of the following filter options:
  • Overwrite
    . Overwrites the existing file.
  • Append Timestamp
    . Retains the existing file and appends a timestamp to the name of file being transferred.
S3 Accelerated Transfer*
Select whether to use Amazon S3 Transfer Acceleration on the S3 bucket. To use Transfer Acceleration, accelerated transfer must be enabled for the bucket. Select one of the following options:
  • Disabled
    . Do not use Amazon S3 Transfer Acceleration.
  • Accelerated
    . Use Amazon S3 Transfer Acceleration.
  • Dualstack Accelerated
    . Use Amazon S3 Transfer Acceleration on a dual-stack endpoint.
Minimum Upload Part Size*
Minimum upload part size in megabytes when uploading a large file as a set of multiple independent parts. Use this option to tune the file load to Amazon S3.
Multipart Upload Threshold*
Multipart download minimum threshold in megabytes to determine when to upload objects in multiple parts in parallel.
*Not applicable when you read data from Databricks Delta.

0 COMMENTS

We’d like to hear from you!