Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Bolo.

Table of Contents

Search

  1. Introducing Mass Ingestion
  2. Getting Started with Mass Ingestion
  3. Connectors and Connections
  4. Mass Ingestion Applications
  5. Mass Ingestion Databases
  6. Mass Ingestion Files
  7. Mass Ingestion Streaming
  8. Monitoring Mass Ingestion Jobs
  9. Asset Management
  10. Troubleshooting

Mass Ingestion

Mass Ingestion

Database Ingestion service properties

Database Ingestion service properties

To change or optimize the behavior of the Database Ingestion service that your Secure Agent group uses, configure Database Ingestion properties for your runtime environment.
To configure the properties, open your runtime environment and click
Edit
. Under
System Configuration Details
, select the
Database Ingestion
service and the
DBMI_AGENT_CONFIG
type.
The following table describes the Database Ingestion agent service properties:
Property
Description
maxTaskUnits
The maximum number
application ingestion
tasks and
database ingestion
tasks that can run concurrently on an on-premises machine where the Secure Agent is running.
Each ingestion task requires at least 1 task unit to run.
Informatica recommends that you configure no more than 2.5 task units per core. For example, if you have an 8-core machine, you can set the maxTaskUnits property to 20. If you have a 16-core machine, you can set it to 40.
During initial load processing, this property determines the number of tables that can be unloaded simultaneously. Remaining tables are queued and start unload processing when resources become available.
During incremental load processing, this property determines the number of
application ingestion
and
database ingestion
jobs that can run simultaneously.
A single job can process many tables. The total number of tables that can be processed is limited only by available memory. On the average, 25 MB of RAM is required per table for an initial load task based on a 1 KB row size.
serviceLogRetentionPeriod
The number of days to retain each internal Database Ingestion service log file after the last update is written to the file. When this retention period elapses, the log file is deleted. The default value is 7 days.
Service logs are retained on the Secure Agent host where they are created:
<infaagent>/apps/Database_Ingestion/logs
.
This property is applicable to both
Mass Ingestion Applications
and
Mass Ingestion Databases
.
taskLogRetentionPeriod
The number of days to retain each job log file after the last update is written to the file. When this retention period elapses, the log file is deleted. The default value is 7 days.
ociPath
For Oracle sources and targets, the path to the Oracle Call Interface (OCI) file oci.dll or libcIntsh.so. For a DBMI agent that is running, this value is appended to the path that is specified in the PATH environment variable on Windows or in the LD_LIBRARY_PATH environment variable on Linux.
This property is applicable only to
Mass Ingestion Databases
.
serviceUrl
The URL that the Database Ingestion service uses to connect to the
Informatica Intelligent Cloud Services
cloud.
This property is applicable to both
Mass Ingestion Applications
and
Mass Ingestion Databases
.
logLevel
The level of detail to include in the logs that the Database Ingestion service produces. Options are:
  • TRACE
  • DEBUG
  • INFO
  • WARN
  • ERROR
The default value is TRACE.
This property is applicable to both
Mass Ingestion Applications
and
Mass Ingestion Databases
.
taskExecutionHeapSize
The maximum heap size, in gigabytes, for the Task Execution service. This value, in conjunction with maxTaskUnits property, affects the number of concurrent
application ingestion
and
database ingestion
tasks that can run on a Secure Agent. Try increasing the heap size to run more tasks concurrently. Enter this value followed by "g" for gigabytes, for example, '9g'. The default value is '8g'.
This property is applicable to both
Mass Ingestion Applications
and
Mass Ingestion Databases
.
useProxy
Set this property to true to enable the DBMI Agent to go through a proxy when connecting to or writing data to targets. The DBMI Agent then uses the proxy settings from the Secure Agent proxy configuration. By default, proxy settings are not used.
This property is applicable to both
Mass Ingestion Applications
and
Mass Ingestion Databases
.
intermediateStorageDirectory
For incremental load and combined initial and incremental load jobs, the local root directory under which intermediate files that contain data are stored when the
Enable Persistent Storage
option is selected in the associated task definitions.
This property is applicable only to
Mass Ingestion Databases
.
storageBackupDirectory
For incremental load and combined initial and incremental load jobs, the path to the directory that stores backup files when the
Enable Persistent Storage
option is selected in the associated task definitions.
This property is applicable only to
Mass Ingestion Databases
.
storageProperties
For incremental load and combined initial and incremental load jobs, a comma-separated list of key=value pairs that is used when the
Enable Persistent Storage
option is selected in the associated task definitions. Specify this property only at the direction of Informatica Global Customer Support.
This property is applicable only to
Mass Ingestion Databases
.
task_container.jvm.allowExceptionForInvalidEncodedData
If you receive transliteration errors that report invalid encoding to UTF-8, and you do not want to repair or correct the source data, set this property to false so that database ingestion jobs do not fail when trying to unload the data from the source. With this setting, the Database Ingestion service passes an equivalent Java property to the DataDirect JDBC driver to prevent the exception from occurring. After you set this property, you must restart the Database Ingestion service.
This property is applicable only to
Mass Ingestion Databases
.
testProperty
Do not set this property. It is intended for internal use by Informatica Global Customer Support and technical staff. This property appears only if you select DBMI_AGENT_ENV in the
Type
field.

0 COMMENTS

We’d like to hear from you!