Read from Microsoft Azure Data Lake Storage Gen2 and write to Databricks Delta
Read from Microsoft Azure Data Lake Storage Gen2 and write to Databricks Delta
You can configure pushdown optimization for a mapping that uses an Microsoft Azure Data Lake Storage Gen2 connection in the Source transformation to read from Microsoft Azure Data Lake Storage Gen2 and a Databricks Delta connection in the Target transformation to write to Databricks Delta.
Example
You want to load data from an Microsoft Azure Data Lake Storage Gen2 based storage object to Databricks Delta for analytical purposes. You want to transform the data before it is made available to users. Use an Microsoft Azure Data Lake Storage Gen2 connection to read data from a Microsoft Azure Data Lake Storage Gen2 source and a Databricks Delta connection to write to a Databricks Delta target. Configure full pushdown optimization in the mapping task to optimize the performance of loading data to Databricks Delta. Pushdown optimization enhances the performance of the task and reduces the cost involved.