Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.
You were redirected to the closest matching page. If this looks correct, please consider bookmarking it for future reference.
Not Found
English

Page Not Found

The page or file you are looking for has been moved or no longer exists.

The following links might help you find the content you need:


You can also search the Documentation Portal or visit the home page.

English
  • Data Integration Connectors
  • All Products

Table of Contents

Search

  1. Introduction to Amazon S3 V2 Connector
  2. Amazon S3 V2 connections
  3. Amazon S3 V2 sources and targets
  4. Mappings and mapping tasks with Amazon S3 V2
  5. Migrating a mapping
  6. Upgrading to Amazon S3 V2 Connector
  7. Data type reference
  8. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Back Next

Recursively read files from directories

Recursively read files from directories

You can read objects stored in subdirectories in Amazon S3 V2 mappings in advanced mode. You can use recursive read for flat, Avro, JSON, ORC, and Parquet files. The files that you read using recursive read must have the same metadata.
To enable recursive read, select the source type as
Directory
 in the advanced source properties. Enable the
Recursive Directory Read
 advanced source property to read objects stored in subdirectories.
You can also use recursive read when you
specify wildcard characters
 in a folder path or file name. For example, you can use a wildcard character to recursively read files in the following ways:
  • Folder path is
    /abc*/
    . Returns all files within any folder or subfolder that has a pattern
    abc
     at the starting of the folder name.
  • Folder path is
    /abc*/
     and file name is
    myfile*
    . Returns all files that have a pattern
    myfile
     at the starting of the file name within a folder or subfolder and has a pattern
    abc
     at the starting of the folder name.
When there are no files available in the folder path of the source directory and the files are present only in the subdirectory within a parent directory, the mapping fails because there are no files to override in the source directory. However, when you place some files in overridden source at the parent directory path, the mapping is successful and picks data from the overridden path.
0 COMMENTS
We’d like to hear from you!