Hi, I'm Ask INFA!
What would you like to know?
ASK INFAPreview
Please to access Ask INFA.

Table of Contents

Search

  1. Preface
  2. Introduction to Databricks Delta Connector
  3. Connections for Databricks Delta
  4. Mappings and mapping tasks with Databricks Delta connector
  5. Databricks Delta pushdown optimization (SQL ELT)
  6. Data type reference

Databricks Delta Connector

Databricks Delta Connector

Override the update operation

Override the update operation

You can specify an update override to override the update query that the Secure Agent generates for the update operation.
When you configure an update override, the Secure Agent uses the query that you specify, stages the data in files, and then loads that data into a temporary table using the merge command. The data from the temporary table is then loaded to the Databricks target table. The syntax that you specify for the update query must be supported by Databricks Delta:
Specify the update override query in the following format:
MERGE INTO <Target table name> AS A USING :TU AS B ON A.<Column1> = B.<Column1> WHEN MATCHED THEN UPDATE SET A.<Column2> = B.<Column2>, A.<Column3>= B.<Column3> ... A.<ColumnN>= B.<ColumnN>
where,
:TU
represents the incoming data source for the target transformation. The Secure Agent replaces
:TU
with a temporary table name while running the mapping and does not validate the update query.
While specifying the name of the target table, you must provide a fully qualified name in the format:
<Catalog_Name>.<Database_Name>.<Target_TableName>
.
When you configure an update override in a mapping to write to Databricks Delta, consider the following rules and guidelines:
  • Ensure that the column names for :TU matches the target table column names.
  • Specify the update query with a valid SQL syntax because Databricks Delta Connector replaces :TU with a temporary table name and does not validate the update query.
  • Do not change the order of the column in the mappings when you configure the update override option.
  • The column names used in the override query must not be unconnected in the target field mapping.
  • You can only override a single SQL query at runtime.
  • You cannot perform an update override on insert, upsert, delete, data driven, and dynamic schema handling operations.

0 COMMENTS

We’d like to hear from you!