Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Bolo.

Table of Contents

Search

  1. Introducing Mass Ingestion
  2. Getting Started with Mass Ingestion
  3. Connectors and Connections
  4. Mass Ingestion Applications
  5. Mass Ingestion Databases
  6. Mass Ingestion Files
  7. Mass Ingestion Streaming
  8. Monitoring Mass Ingestion Jobs
  9. Asset Management
  10. Troubleshooting

Mass Ingestion

Mass Ingestion

Amazon S3 V2 connection properties

Amazon S3 V2 connection properties

When you set up an Amazon S3 V2 connection, you must configure the connection properties.
The following table describes the Amazon S3 V2 connection properties:
Property
Description
Connection Name
Name of the connection.
The name is not case sensitive and must be unique within the domain. You can change this property after you create the connection. The name cannot exceed 128 characters, contain spaces, or contain the following special characters:
~ ` ! $ % ^ & * ( ) - + = { [ } ] | \ : ; " ' < , > . ? /
Description
Optional. Description of the connection.
The description cannot exceed 4,000 characters.
Type
The Amazon S3 V2 connection type.
Runtime Environment
Name of the runtime environment where you want to run the tasks.
You cannot run a database ingestion task on a Hosted Agent or serverless runtime environment.
Access Key
Access key to access the Amazon S3 bucket. Provide the access key value based on the following authentication methods:
  • Basic authentication. Provide the actual access key value.
  • IAM authentication.
    Do not provide the access key value.
    .
  • Temporary security credentials via assume role. Provide the secret access key of an IAM user with no permissions to access Amazon S3 bucket.
  • Assume role for EC2.
    Do not provide the access key value.
  • Credential profile file authentication.
    Do not provide the access key value.
  • Federated user single sign-on. Do not provide the secret access key value.
Secret Key
Secret access key to access the Amazon S3 bucket.
The secret key is associated with the access key and uniquely identifies the account. Provide the secret access key value based on the following authentication methods:
  • Basic authentication. Provide the actual access secret value.
  • IAM authentication.
    Do not provide the access secret value.
  • Temporary security credentials via assume role. Provide access secret of an IAM user with no permissions to access Amazon S3 bucket.
  • Assume role for EC2.
    Do not provide the access key value.
  • Credential profile file authentication.
    Do not provide the access secret value.
  • Federated user single sign-on. Do not provide the access secret value.
IAM Role ARN
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role assumed by the user to use the dynamically generated temporary security credentials.
Enter the value of this property if you want to use the temporary security credentials to access the AWS resources.
This property is not applicable to an application ingestion task and database ingestion task.
Even if you remove the IAM role that enables the agent to access the Amazon S3 bucket, and create a connection, the test connection is successful.
For more information about how to obtain the ARN of the IAM role, see the AWS documentation.
External Id
Optional. Specify the external ID for a more secure access to the Amazon S3 bucket when the Amazon S3 bucket is in a different AWS account.
Use EC2 Role to Assume Role
Optional. Select the check box to enable the EC2 role to assume another IAM role specified in the IAM Role ARN option.
The EC2 role must have a policy attached with a permission to assume an IAM role from the same or different account.
By default, the Use EC2 Role to Assume Role check box is not selected.
Folder Path
Bucket name or complete folder path to the Amazon S3 objects.
Do not use a slash at the end of the folder path. For example:
<bucket name>/<my folder name>
.
Master Symmetric Key
Optional. Provide a 256-bit AES encryption key in the Base64 format when you enable client-side encryption. You can generate a key using a third-party tool.
Not applicable for an application ingestion task, database ingestion task, or streaming ingestion task.
Customer Master Key ID
Optional. Specify the customer master key ID or alias name generated by AWS Key Management Service (AWS KMS) or the Amazon Resource Name (ARN) of your custom key for cross-account access.
You must generate the customer master key for the same region where Amazon S3 bucket resides. You can specify the following master keys:
Customer generated customer master key
Enables client-side or server-side encryption.
Default customer master key
Enables client-side or server-side encryption. Only the administrator user of the account can use the default customer master key ID to enable client-side encryption.
Not applicable for an application ingestion task, database ingestion task, or streaming ingestion task.
S3 Account Type
The type of the Amazon S3 account.
Select from the following options:
  • Amazon S3 Storage. Enables you to use the Amazon S3 services.
  • S3 Compatible Storage. Specify the endpoint for a third-party storage provider such as Scality RING or MinIO.
By default, Amazon S3 storage is selected.
REST Endpoint
The S3 storage endpoint.
Specify the S3 storage endpoint in HTTP or HTTPs format when you select the S3 compatible storage option. For example,
http://s3.isv.scality.com
Region Name
The AWS region of the bucket that you want to access.
Select one of the following regions:
  • Asia Pacific (Mumbai)
  • Asia Pacific (Seoul)
  • Asia Pacific (Singapore)
  • Asia Pacific (Sydney)
  • Asia Pacific (Tokyo)
  • Asia Pacific (Hong Kong)
  • AWS GovCloud (US)
  • AWS GovCloud (US-East)
  • Canada (Central)
  • China (Beijing)
  • China (Ningxia)
  • EU (Ireland)
  • EU (Frankfurt)
  • EU (London)
  • EU (Paris)
  • EU (Stockholm)
  • South America (Sao Paulo)
  • Middle East (Bahrain)
  • US East (Ohio)
  • US East (N. Virginia)
  • US West (N. California)
  • US West (Oregon)
Default is US East (N. Virginia).
Federated SSO IdP
SAML 2.0-enabled identity provider for the federated user single sign-on to use with the AWS account. Amazon S3 V2 connector supports only
ADFS 3.0
identity provider. Select
None
if you do not want to use federated user single sign-on.
Federated user single sign-on is not applicable to application ingestion tasks, database ingestion tasks, and streaming ingestion tasks.
Other Authentication Type
Select one the following authentication types:
  • NONE
  • Credential Profile File Authentication
Select the Credential Profile File Authentication option to access the Amazon S3 credentials from a credential file that contains the access key and secret key.
Specify the credential profile file path and the profile name to establish the connection with Amazon S3.
You can use permanent IAM credentials or temporary session tokens when you configure the Credential Profile File Authentication.
Default is NONE.
Credential Profile File Path
Optional. Specify the credential profile file path.
If you do not specify the credential profile path, the Secure Agent uses the credential profile file present in the following default location in your home directory:
~/.aws/credentials
Mass Ingestion Databases
has not been certified with the
Credential Profile File Path
and
Profile Name
connection properties.
Mass Ingestion Databases
finds AWS credentials by using the default credential provider chain that is implemented by the DefaultAWSCredentialsProviderChain class, which includes the credential profile file.
Profile Name
Optional. Name of the profile in the credential profile file used to get the credentials.
If you do not specify the profile name, the credentials from the default profile in the credential profile file are used.

Federated user single sign-on connection properties

Configure the following properties when you select
ADFS 3.0
in
Federated SSO IdP
:
Property
Description
Federated User Name
User name of the federated user to access the AWS account through the identity provider.
Federated User Password
Password for the federated user to access the AWS account through the identity provider.
IdP SSO URL
Single sign-on URL of the identity provider for AWS.
Not applicable for a streaming ingestion task.
SAML Identity Provider ARN
ARN of the SAML identity provider that the AWS administrator created to register the identity provider as a trusted provider.
Role ARN
ARN of the IAM role assumed by the federated user.

0 COMMENTS

We’d like to hear from you!