Auto-Pilot Monitoring enhances the provisioning ability of Lakehouse Optimizer in order to continually monitor on jobs and clusters that undergo a configuration change or are shared-compute
What Auto-Pilot does
provisions automatically previously monitored assets
provisions automatically new assets with shared-compute policies
assists manual provisioning for assets that are owned by others
automatically upgrades Telemetry Agent on LHO version upgrade
Automatic Provisioning
At each LHO restart and every 1h schedule, the Auto Pilot does the following when
if Global Script ENABLED
enables monitoring on all jobs that contain at least one task with a shared-compute cluster as execution engine
enables monitoring on all clusters that are using shared-compute as access policy
all other jobs and clusters maintain their monitor status based on global init script option
if Global Script DISABLED
enables monitoring on all jobs and clusters that contain were previously monitored (with an older LHO configuration)
e.g. the user changes the Secret Scope name and affects the monitoring status. Auto-Pilot reenables monitoring in this scenario
Assisting User-Triggered Provisioning
Auto-Pilot also allows admin users to enable monitoring on All-Purpose-Clusters or Jobs running on All-Purpose-Clusters that are owned by other users.
In this scenario, when provisioning is triggered manually by the user, Auto-Pilot temporary changes the ownership of the job or cluster to the user that triggered manually the provisioning and after the configuration is updated to reference the LHO Telemetry Agent, the ownership is set back to the original owner.
Ownership change is also executed when Auto-Pilot is triggered at restart or on a schedule basis. In this scenario, the new temporary owner is set to the Service Principal used by LHO.
Automatic Telemetry Agent Update
When a new version of LHO is available, on upgrade the Auto Pilot will update the LHO Telemetry Agent in each published workspace for all assets that were previously monitored or are of shared-compute
type.
Please see section Automatic Provisioning for more details.
How to enable/disable Auto-Pilot
env variables
AUTO_PILOT_ENABLED
= true/falsedefault: true
0 Comments