Table of Contents

Search

  1. Introduction to Databricks Delta Connector
  2. Connections for Databricks Delta
  3. Mappings and mapping tasks with Databricks Delta connector
  4. Databricks Delta pushdown optimization (SQL ELT)
  5. Data type reference

Databricks Delta Connector

Databricks Delta Connector

Pushdown optimization types

Pushdown optimization types

When you apply pushdown optimization, the task pushes transformation logic to the source or target database based on the optimization type you specify in the task properties. Data Integration translates the transformation logic into SQL queries or Databricks Delta commands to the Databricks Delta database. The database runs the SQL queries or Databricks Delta commands to process the transformations.
You can configure the following types of pushdown optimization in a mapping:
None
The task does not push down the transformation logic to the Databricks Delta database.
Full
The task pushes as much of the transformation logic as possible to process in the Databricks Delta target database.
Data Integration analyses all the transformations from the source to the target. If all the transformations are compatible in the target, it pushes the entire mapping logic to the target. If it cannot push the entire mapping logic to the target, Data Integration first pushes as much transformation logic to the source database and then pushes as much transformation logic as possible to the target database.
When a transformation is not supported in the mapping, the task partially pushes down the mapping logic to the point where the transformation is supported for pushdown optimization. However, this applies only to Databricks SQL endpoints.
When you enable full pushdown optimization, you can determine how Data Integration handles the job when pushdown optimization does not work. You can use the If pushdown mode is not possible, cancel the task option to set the task to fail or run without pushdown optimization.
Source
The task pushes as much as the transformation logic as possible to process in the Databricks Delta source database. This applies only to Databricks SQL endpoints.
You cannot enable source push down optimization for mappings in advanced mode.

0 COMMENTS

We’d like to hear from you!
Hi, I'm Bolo!
What would you like to know?
Please to access Bolo.