Mass Ingestion

Back Next

Troubleshooting an application ingestion task

If you change the unsupported data type of a source field to a supported data type, the change might not be replicated to the target.: This problem occurs when the
Modify field
schema drift option is set to
Replicate
and the
Add field
option is set to
Ignore
.

When you deploy an
application ingestion
job,
Mass Ingestion Applications
does not create target columns for the source fields of unsupported data types. If you change the unsupported data type to a supported data type for the source field,
Mass Ingestion Applications
processes the modification made to the source field but does not replicate the change to the target. When
Mass Ingestion Applications
tries to add a field with the supported data type to the target, the operation is ignored because the schema drift option
Add column
is set to
Ignore
.

To handle this situation, perform the following steps:

On the
Schedule and Runtime Options
page in the
application ingestion
task wizard, in the
Schema Drift Options
section, set the
Add field
option to
Replicate
.
Change the source field data type to a supported data type again so that the
application ingestion
job can detect the schema change.
The
application ingestion
job does not propagate the field values that were added prior to changing the data type of the source field.

If you want to propagate all the values from the source field to the target, resynchronize the target table with the source.
If you change a primary key constraint on the source, Mass Ingestion Applications stops processing the source object on which the DDL change occurred.: This problem occurs if you add or drop a primary key constraint, or if you add or drop a field from an existing primary key.

To resume processing the source object for combined initial and incremental jobs, resynchronize the target table with the source.

To resume processing the source object for incremental jobs, perform the following steps:

On the
Source
page of the
application ingestion
task wizard, add an object selection rule to exclude the source object.
Redeploy the task.
Mass Ingestion Applications
deploys the modified task and deletes the information about the primary keys of the excluded object.

Edit the task again to delete the object selection rule that excluded the source object.
Redeploy the task.
An application ingestion initial load job that processes many source objects and has a Google BigQuery target might fail with the following error:: The job has timed out on the server. Try increasing the timeout value.

This problem occurs when the job is configured to process many source objects and the connection configured for the Google BigQuery target times out before completing the processing of the source objects. To resolve this issue, you must increase the timeout interval for the Google BigQuery V2 connection configured for the target.

To increase the timeout interval for the connection, perform the following steps:

In Administrator, open the Google BigQuery V2 connection associated with the
application ingestion
job in edit mode.
In the
Provide Optional Properties
field, set the timeout property to the required timeout interval in seconds. Use the following format:
"timeout": "<timeout_interval_in_seconds>"

Save the connection.
Redeploy the
application ingestion
job.

Rename Saved Search

Table of Contents

Mass Ingestion

Mass Ingestion

Troubleshooting an application ingestion task

Troubleshooting an application ingestion task