Advanced Clusters

Back Next

Learn about resource access

To process data, the Secure Agent and the self-service cluster access the resources that are part of a job, including resources on the cloud platform, source and target data, and staging and log locations.

The agent and the cluster access resources to perform the following tasks:

Design a mapping.

Connect to a self-service cluster.

Run a job, including a data preview job.

Poll logs.

Designing a mapping: When you design a mapping, the Secure Agent accesses sources and targets so that you can read and write data.

For example, when you add a Source transformation to a mapping, the Secure Agent accesses the source to display the fields that you can use in the rest of the mapping. The Secure Agent also accesses the source when you preview data.

To access a source or target, the Secure Agent uses the connection properties. For example, the Secure Agent might use the user name and password that you provide in the connection properties to access a database.
Connecting to a self-service cluster: To connect to a self-service cluster, the Secure Agent stores cluster information in the staging location.
Running a job: To run an
advanced job
, the Secure Agent and worker nodes access sources and targets, the staging location, and the log location.
The following image shows how resources are accessed when a job runs on a self-service cluster:

The following steps describe the process that the Secure Agent and cluster nodes use to run the job:
You run a job.
The Secure Agent stores cluster information in the staging location using the permission from the Secure Agent role.
The Secure Agent accesses the cluster using the kubeconfig file and submits a job to the self-service cluster.
The worker nodes use the access information to the staging and log storage accounts to get job dependencies from the staging location, to stage data in the staging location, and to store logs in the log location.
The Secure Agent uses the Secure Agent role to upload the agent job log to the log location.
Polling logs: When you use Monitor, the Secure Agent accesses the log location to poll logs.
To poll logs from the log location, the Secure Agent uses the role that is assigned to the Secure Agent machine.

Rename Saved Search

Table of Contents

Advanced Clusters

Advanced Clusters

Learn about resource access

Learn about resource access