Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.
You were redirected to the closest matching page. If this looks correct, please consider bookmarking it for future reference.
Not Found
English

Page Not Found

The page or file you are looking for has been moved or no longer exists.

The following links might help you find the content you need:


You can also search the Documentation Portal or visit the home page.

English
  • Data Integration Connectors
  • All Products

Table of Contents

Search

  1. Introduction to Amazon S3 V2 Connector
  2. Amazon S3 V2 connections
  3. Amazon S3 V2 sources and targets
  4. Mappings and mapping tasks with Amazon S3 V2
  5. Migrating a mapping
  6. Upgrading to Amazon S3 V2 Connector
  7. Data type reference
  8. Troubleshooting

Amazon S3 V2 Connector

Amazon S3 V2 Connector

Back Next

Custom manifest file

Custom manifest file

You can read multiple files, which are of flat format type, from Amazon S3
and write data to a target
. To read multiple flat files, all files must be available in the same Amazon S3 bucket.
When you want to read from multiple sources in the Amazon S3 bucket, you must create a
.manifest
 file that contains all the source files with the respective absolute path or directory path. You must specify the
.manifest
 file name in the following format:
<file_name>.manifest
.
For example, the
.manifest
 file contains source files in the following format: 

{
	"fileLocations": 
	[
		{
			"URIs":
			[
				"dir1/dir2/dir3/file_1.csv",				
				"dir1/dir2/dir3/file_2.csv",
				"dir1/file_3.csv"
			]
		}, 	
		{
			"URIPrefixes": 
			[
				"dir1/dir2/dir3/",
				"dir1/dir2/dir4/"
			]
		},
		{
			"WildcardURIs": 
			[
				"dir1/dir2/dir3/*.csv"
			]
		}
	],                                       
	"settings": 
	{
		"stopOnFail": "true"
	}
}
The custom manifest file contains the following tags:
  • URIs. Specify the path for the files relative to the bucket name.
  • URIPrefixes. Specify the path for the directory relative to the bucket name.
  • WildcardURIs. Specify an asterisk (*) wildcard in the file name, which are of flat format type, to fetch files from the Amazon S3 bucket. Specify the asterisk (*) wildcard to fetch all the files or only the files that match the name pattern.
You can specify URIs, URIPrefixes, WildcardURIs, or all sections within fileLocations in the
.manifest
 file.
You cannot use the wildcard characters to specify folder names. For example, { "WildcardURIs": [ "multiread_wildcard/dir1*/", "multiread_wildcard/*/" ] }.
The
Data Preview
 tab displays the data of the first file available in the URI specified in the
.manifest
 file. If the URI section is empty, the first file in the folder specified in URIPrefixes is displayed.
0 COMMENTS
We’d like to hear from you!