| Info |
|---|
Auto-Pilot Monitoring enhances the provisioning ability of Lakehouse Optimizer in order to continually monitor on jobs and clusters that undergo a configuration change or are shared-computeautomatically monitor ALL workloads in all published Databricks workspaces |
| Table of Contents |
|---|
What Auto-Pilot does
...
1. Automatic Provisioning
At each LHO restart and every 1h (configurable schedule), the Auto Pilot does the following when
if Global Init Script is ENABLED
enables monitoring on all workloads that contain at least one task with Shared Access Mode cluster (cluster configuration:
"data_security_mode": "USER_ISOLATION")all other jobs and clusters maintain their monitor status based on global init script option
...
if Global Init Script is DISABLED
enables monitoring on all workloads by enabling Cluster-Scoped Init Scripts and individual cluster Spark configurations , using Secret Scopes
Auto-Pilot also allows admin users to enable monitoring on All-Purpose-Clusters that are owned by other users.
...
Temporary ownership change is executed when Auto-Pilot is triggered at restart or on a schedule basis.
...
The new temporary owner is set to the Service Principal used by LHO until the workload configuration is updated to reference the LHO Telemetry Agent. When configuration update is finished, the owner is set back to the original owner.
2. Automatic Telemetry Agent Update
...