Auto-Pilot Monitoring enhances the provisioning ability of Lakehouse Optimizer to automatically monitor ALL workloads in all published Databricks workspaces
What Auto-Pilot does:
1. Automatic Provisioning
At each LHO restart and every 1h schedule, the Auto Pilot does the following when
if Global Init Script ENABLED
enables monitoring on all workloads that contain at least one task with Shared Access Mode cluster
all other jobs and clusters maintain their monitor status based on global init script option
if Global Init Script DISABLED
enables monitoring on all workloads by enabling Cluster-Scoped Init Scripts and individual cluster Spark configurations, using Secret Scopes
Auto-Pilot also allows admin users to enable monitoring on All-Purpose-Clusters that are owned by other users.
Ownership change is executed when Auto-Pilot is triggered at restart or on a schedule basis. In this scenario, the new temporary owner is set to the Service Principal used by LHO.
2. Automatic Telemetry Agent Update
When a new version of LHO is available, on upgrade the Auto Pilot will update the LHO Telemetry Agent in each published workspace.
Please see section Automatic Provisioning for more details.
How to enable/disable Auto-Pilot
env variables
AUTO_PILOT_ENABLED
= true/falsedefault: true
Unity Catalog Support
If you are using Unity Catalog, Auto-Pilot requires additional configuration.
Please follow the following steps:
0 Comments