When you specify SESSSTARTTIME variable in a query in a mapping task to return the Datetime values, specify the query in the following format:
:select to_timestamp('$$$SESSSTARTTIME', 'MM/dd/yyyy HH:mm:ss.SSSSSS') as t;
When you run multiple concurrent mappings to write data to Databricks Delta targets, a transaction commit conflict error might occur and the mappings might fail.
View objects are displayed in the Table panel instead of the View panel while importing a Databricks Delta object. This issue occurs when the Databricks cluster is deployed on AWS cloud service.
To avoid java heap space error when you read or write complex files, set the JVM options for type DTM to increase the -Xms and -Xmx values in the system configuration details of the Secure Agent. The recommended values for -Xms is 512 MB and -Xmx is 1024 MB.
When you import views, the Select Source Object dialog box does not display view objects.
When you test the Databricks Delta connection, the Secure Agent does not validate the values you specify in the Org ID connection parameter.
You cannot use the Hosted Agent as a runtime environment when you configure a mapping to run on the SQL warehouse to read or write data that contains unicode characters.
The number of clusters that the Secure Agent creates to run the mapping depends on the number of Databricks Delta connections used in the transformations in a mapping. For example, if multiple transformations use the same Databricks Delta connection, the mapping runs on a single cluster.
When you keep the mapping designer idle for more than 15 minutes, the metadata fetch throws an exception.
If you change the database name in the connection and run the existing mappings, the mappings start failing. After you change the database name in the connection, you must reimport the objects in the existing mappings before you run the mappings.
Use the following formats to run the mapping successfully, when you import a Databricks Delta source object containing Date or Boolean data types with a simple source filter conditions:
Boolean = 0 or 1
Date = YYYY-MM-DD HH24:MM:SS.US
When you run a mapping with source column data type as string containing
TRUE / FALSE
value and write data to target with Boolean data type column of a Databricks Delta table, the Secure Agent writes data as
0
to the target.
When the Databricks analytics cluster is down and you perform a test connection or import an object, the connection is timed out after 10 minutes.
When you parameterize the source or target connection in a mapping and you do not specify the database name, ensure that you specify the database name in lowercase when you assign a default value for the parameter.
When you parameterize the source filter condition or any expressions in a mapping, ensure that you specify the table name in lowercase when you add the source filter condition or the expression in the
mapping
task. Otherwise, the Secure Agent throws the following exception:
Invalid expression string for filter condition
When you run a mapping to write data to a Databricks Delta target using create target at runtime and the target table already exists, ensure that the target table schema is same. Otherwise, the mapping fails.
When you run a mapping to write data to multiple Databricks Delta targets that use the same Databricks Delta connection and the Secure Agent fails to write data to one of targets, the mapping fails and the Secure Agent does not write data to the remaining targets.
When you use the
Create New at Runtime
option to create a Databricks target, you can parameterize only the target connection and the table name using a parameter file. You cannot parameterize other properties such as
Path
or
DBname
.
The pre-SQL and post-SQL commands run non-linearly. In the session logs, you will see that the target pre-SQL cases are executed before the source pre-SQL queries.
When you run pre-SQL and post-SQL commands to read from sources that have semi-colons within the query, the mappings fails. The queries can only have semi-colons at the end.
When you have unicode data within tables in Databricks Delta objects, configure the property
-Dfile.encoding=UTF-8
in the JVM options.
When you configure a mapping where you have staged data in the Personal Staging Location, the temporary data is not deleted when the mapping stops abruptly. Use
Clean Stop
feature to erase the temporary staged files from the log.
ASK INFAPreview
