What Auto-Pilot does

1. Automatic Provisioning

At each LHO restart and every 1h (configurable schedule), the Auto Pilot does the following

• if Global Init Script is ENABLED

  • enables monitoring on all workloads that contain at least one task with Shared Access Mode cluster (cluster configuration: "data_security_mode": "USER_ISOLATION")

  • all other jobs and clusters maintain their monitor status based on global init script option

Open

Acces Mode: Shared

• if Global Init Script is DISABLED

  • enables monitoring on all workloads by enabling Cluster-Scoped Init Scripts and individual cluster Spark configurations using Secret Scopes

  • Temporary ownership change is executed when Auto-Pilot is triggered at restart or on a schedule basis. The new temporary owner is set to the Service Principal used by LHO until the workload configuration is updated to reference the LHO Telemetry Agent. When configuration update is finished, the owner is set back to the original owner.

2. Automatic Telemetry Agent Update

When a new version of LHO is available, on upgrade the Auto Pilot will update the LHO Telemetry Agent in each published workspace.

Please see section Automatic Provisioning for more details.

How to enable/disable Auto-Pilot

• env variables

  • AUTO_PILOT_ENABLED = true/false

  • default: true

Unity Catalog Support

If you are using Unity Catalog, Auto-Pilot requires additional configuration.

Please follow the following steps:

• Provisioning with Unity Catalog Enabled (AWS)