task to write data to a Databricks Delta target. To create a Databricks Delta target, use the Databricks Delta connection type. The Databricks Delta target requires a Databricks cluster version 6.3 or later.
Databricks Delta is an open source storage layer that provides ACID transactions and works on top of existing data lakes. Databricks uses proprietary Delta software to manage stored data and allow fast access to the data.
You can access Delta Lake tables built on top of the following storage types:
Azure Data Lake Storage (ADLS) Gen2
Amazon Web Services (AWS) S3
The Databricks Delta target writes data to one or more Delta Lake tables on Databricks. You can use the Databricks Delta target in a
streaming ingestion
task for the following use cases:
Ingest bulk data from all streaming sources into Databricks Delta tables
Merge change data capture (CDC) from all streaming sources and write to Databricks Delta tables
The Databricks Delta connection uses a JDBC URL to connect to the Databricks cluster. When you configure the target, you specify the JDBC URL and credentials to use to connect to the cluster. You also define the connection information that the target uses to connect to the staging location in Amazon S3 or Azure Data Lake Storage Gen2.
You specify the tables in Delta Lake to which you want to write the data. The target writes data from record fields to table columns based on matching names.
ASK INFAPreview
