Databricks Delta Connector

Back Next

Introduction to Databricks Delta Connector

You can use Databricks Delta Connector to securely read data from or write data to Databricks Delta.

You can create a Databricks Delta connection and use the connection in mappings and

mapping

tasks. You can use Databricks Delta Connector on the Windows and Linux operating systems.

For Linux operating systems, you can switch mappings to advanced mode to include transformations and functions that enable advanced functionality.

The following section explains how the Secure Agent communicates with Databricks Delta during the design time and runtime:

Design time data flow for mappings and mappings in advanced mode

During the mapping design, the Secure Agent communicates with the Databricks SQL warehouse or Databricks analytics cluster for metadata-related operations.

Runtime data flow for mappings

During the runtime, the Secure Agent communicates with the Databricks SQL warehouse or Databricks data cluster to read or write data.

Only SQL warehouse cluster is applicable for Windows.

The following image shows how the Secure Agent connects to Databricks Delta to read or write data in Data Integration mappings:

The image shows how the Secure Agent connects to Databricks Delta.

The Secure Agent uses Amazon S3 in AWS environment or Azure Data Lake Storage Gen2 in Azure environment for staging the data.

If you use the Databricks SQL warehouse, the Secure Agent starts the Databricks SQL warehouse and then connects to the SQL warehouse to read data from or write data to Databricks Delta tables. When you use Databricks Delta as a source, the Secure Agent runs a SQL query on the Databricks SQL warehouse to read data from a Databricks Delta table. When you use Databricks Delta as a target, the Secure Agent runs a SQL query on the Databricks SQL warehouse to read data from the staging location and write to a Databricks Delta table.

If you use the Databricks cluster, the Secure Agent creates a Databricks cluster to read data from or write data to Databricks Delta tables. When you use Databricks Delta as a source, the Secure Agent runs a spark job in the Databricks cluster to read data from a Databricks Delta table and write to the staging location. When you use Databricks Delta as a target, the Secure Agent runs a spark job in the Databricks cluster to read data from the staging location and write to a Databricks Delta table.