Mass Ingestion

Back Next

Flat File source properties

The following table describes the flat file source properties on the

Source

tab when you define a

streaming ingestion

task:

Property	Description
Connection	Name of the flat file source connection.
Connection Type	The Flat file connection type. The connection type appears automatically based on the connection that you select.
Initial Start Position	Starting position from which the data is to be read in the file to tail. You can choose one of the following positions to start reading: Beginning of File. Read from the beginning of the file to tail. Do not ingest any data that has already been rolled over. Current Time. Read from the most recently updated part of the file to tail. Do not ingest any data that has already been rolled over or any data in the file to tail that has already been written.
Tailing Mode	Tail a file or multiple files based on the logging pattern. You can choose one of the following modes: Single File. Tail only one file. Multiple Files. Tail all the files indicated in the base directory. In this mode, you can enter a regular expression to indicate the files to tail.
File	Absolute path with the name of the file you want to read. Name of the file to tail or regular expression to find the files to tail. Enter the base directory for multiple files mode.

The following table describes the advanced properties that you can configure for flat file sources on the

Source

tab when you define a

streaming ingestion

task:

Connection Property	Description
Rolling Filename Pattern	Name pattern for the file that rolls over. If the file to tail rolls over, the file name pattern is used to identify files that have rolled over. The underlying streaming ingestion Secure Agent recognizes this file pattern. When the Secure Agent restarts, and the file has rolled over, it picks up from where it left off. You can use asterisk () and question mark (?) as wildcard characters to indicate that the files are rolled over in the same directory. For example, ${filename}.log. . Here, asterisk (*) represents the successive version numbers that would be appended to the file name.

Connection Property

Description

Rolling Filename Pattern

Name pattern for the file that rolls over.

If the file to tail rolls over, the file name pattern is used to identify files that have rolled over. The underlying

streaming ingestion

Secure Agent recognizes this file pattern. When the Secure Agent restarts, and the file has rolled over, it picks up from where it left off.

You can use asterisk (*) and question mark (?) as wildcard characters to indicate that the files are rolled over in the same directory. For example,

${filename}.log.*

. Here, asterisk (*) represents the successive version numbers that would be appended to the file name.