What Auto-Pilot does

• provisions automatically previously monitored assets

• provisions automatically new assets with shared-compute policies

• assists manual provisioning for assets that are owned by others

• automatically upgrades Telemetry Agent on LHO version upgrade

1. Automatic Provisioning

At each LHO restart and every 1h (configurable schedule), the Auto Pilot does the following when

• if Global Init Script is ENABLED

  • enables monitoring on all jobs workloads that contain at least one task with a shared-compute cluster as execution engine

  • enables monitoring on all clusters that are using shared-compute as access policy

  • all Shared Access Mode cluster (cluster configuration: "data_security_mode": "USER_ISOLATION")

  • all other jobs and clusters maintain their monitor status based on global init script option

...

• if Global Init Script is DISABLED

  • enables monitoring on all jobs and clusters that contain were previously monitored (with an older LHO configuration)

    • e.g. the user changes the Secret Scope name and affects the monitoring status. Auto-Pilot reenables monitoring in this scenario

Assisting User-Triggered Provisioning

Auto-Pilot also allows admin users to enable monitoring on All-Purpose-Clusters or Jobs running on All-Purpose-Clusters that are owned by other users.

In this scenario, when provisioning is triggered manually by the user, Auto-Pilot temporary changes the ownership of the job or cluster to the user that triggered manually the provisioning and after the configuration is updated to reference the LHO Telemetry Agent, the ownership is set back to the original owner.

...

• • workloads by enabling Cluster-Scoped Init Scripts and individual cluster Spark configurations using Secret Scopes

  • Temporary ownership change is executed when Auto-Pilot is triggered at restart or on a schedule basis.

...

• • The new temporary owner is set to the Service Principal used by LHO until the workload configuration is updated to reference the LHO Telemetry Agent. When configuration update is finished, the owner is set back to the original owner.

2. Automatic Telemetry Agent Update

When a new version of LHO is available, on upgrade the Auto Pilot will update the LHO Telemetry Agent in each published workspace for all assets that were previously monitored or are of shared-compute type.

Please see section Automatic Provisioning for more details.

How to enable/disable Auto-Pilot

• env variables

  • AUTO_PILOT_ENABLED = true/false

  • default: true

Unity Catalog Support

If you are using Unity Catalog, Auto-Pilot requires additional configuration.

Please follow the following steps:

• Provisioning with Unity Catalog Enabled (AWS)