Mass Ingestion

Back Next

Hadoop Files V2 source properties

When you define a file ingestion task with an Hadoop Files V2 source, you must enter source options on the

Source

tab of the task wizard. The options vary based on the file pickup method that you select for the task.

You can overwrite the file name pattern, folder, and table parameters, and define your own variable for sources by using the job resource of the Mass Ingestion Files REST API. For more information, see Mass Ingestion Files REST API.

The following table describes the source options:

Option	Description
File Pickup	The file ingestion task supports the following file pickup methods: By Pattern . The file ingestion task picks up files by pattern. By File List . The file ingestion task picks up files based on a file list.
Source Directory	Directory from where files are transferred.
Add Parameters	Create an expression to add it as a Source Directory parameter. For more information, see Add Parameters.
Include files from sub folders	This applies when File Pickup is By Pattern . Transfer files from all subfolders under the defined source directory.
File Pattern	This applies when File Pickup is By Pattern . File name pattern used to select the files to transfer. Based on the file pattern that you have selected, enter the file name patterns. Select one of the following file patterns: Wildcard. Use the following wildcard character filters: An asterisk () to represent any number of characters. A question mark (?) to represent a single character. Regex. Use regular expression to match the pattern type. Consider the following samples: ^(?!.(?:out\|baz\|foo)).$ all except Identifies all files except for files whose name contains out, foo, and baz. ([a-zA-Z0-9\s_\\.\-:])+(.doc\|.docx\|.pdf)$ Identifies all files that have an extension of doc, docx, or pdf. ^(?!out).\.txt$ Identifies all text files except for files whose name contains out.txt .
File Date	This applies when File Pickup is By Pattern . A date and time expression for filtering the files to transfer. Select one of the following options: Greater than or Equal . Filters files that are modified on or after the specified date and time. To specify a date, click the calendar. To specify a time, click the clock. Less than or Equal . Filters files that are modified before or on the specified date and time. Equal . Filters files that are modified on the specified date and time. Click the calendar to select the date and the clock to select the time. Days before today . Filters files that are modified within the specified number of days until the current date (today). Enter the number of days. The current date calculation starts from 00:00 hours. For example, if you schedule the file ingestion task to run weekly and want to filter for the files that were modified in the previous week, set Days before today to 7. The task will pick up any file with a date between 7 days ago and the date on which it runs.
Time Zone	This applies when File Pickup is By Pattern . If you selected a File Date option, enter the time zone of the location where the files are located.
File Size	This applies when File Pickup is By Pattern . Filters the files to transfer based on file size. Enter the file size, select the file size unit, and filter options. Select one of the following filter options: Greater than or Equal . Filters files that are greater than or equal to the specified size. Less than or Equal . Filters files that are less than or equal to the specified size. Equal . Filters files that have the specified size.
The file path containing the list of files	This applies when File Pickup is By File List . Select this option to provide the path that contains the list of files to pick up and enter the file path.
File list	This applies when File Pickup is By File List . Select this option to provide the list of files to pick up and enter a comma-separated list of file names.
Skip Duplicate Files	Indicates whether to skip duplicate files. If you select this option, the file ingestion task does not transfer files that have the same name and creation date as another file. The file ingestion task marks these files as duplicate in the job log. If you do not select this option, the task transfers all files, even files with duplicate names and creation dates.
Check file stability	Indicates whether to verify that a file is stable before a file ingestion task attempts to pick it. The task skips unstable files it detects in the current run.
Stability check interval	This applies when you enable the Check file stability option. Time in seconds that a file ingestion task waits to check the file stability. For example, if the stability time is 15 seconds, the file ingestion task detects all the files in the source folder that match the defined file pattern, waits for 15 seconds, and processes only the stable files. The interval ranges between 10 seconds to 300 seconds. Default is 10 seconds.
Batch Size	The number of files a file ingestion task can transfer in a batch. Default is 5.

Rename Saved Search

Table of Contents

Mass Ingestion

Mass Ingestion

Hadoop Files V2 source properties

Hadoop Files V2 source properties