Compute with Shared Access mode are only available when Unity Catalog is enabled for the workspace. Global Init Scripts are not executed for shared access mode cluster, which means the LHO can only enable monitoring for them by means of Cluster-Scoped Init Scripts stored in Unity Catalog volumes.

Moreover, loading init scripts from UC volumes is only supported in Databricks Runtimes 13.3 and more recent.

Configure Cluster Scope Init Script Storage

cluster-init-script - is a LHO script that copies the LHO Telemetry Agent jar from DBFS to the storage area accessible by clusters, which can be either in /Volumes or in /Workspace (workspace files) depending on Unity Catalog setup.

Unity Catalog Enabled

At the moment of this documentation creation, Databricks does not offer an API to upload the cluster-init-script in /Volumes, therefore the admin needs to configure the cluster-init-script by following the next steps.

Step 1) Use the following link to download the LHO cluster-init-script

• https://bplmdemoappstg.blob.core.windows.net/libraries/LHO_VERSION/init_script.sh

  • e.g. https://bplmdemoappstg.blob.core.windows.net/libraries/2.7.2/init_script.sh

Step 2) Prepare storage path in Catalog Explorer

• open Catalog Explorer

• select schema: main/default

• create volume: bp-lakehouse-monitor

Step 3) Upload script to

/Volumes/main/default/bp-lakehouse-monitor/script/

The complete path should look like:

/Volumes/main/default/bp-lakehouse-monitor/script/init_script.sh

Step 4) In Databricks

• open Catalog

• open Metastore details

• open Allowed JARs/Init Scripts

• add a new entry for

/Volumes/main/default/bp-lakehouse-monitor/script/init_script.sh

Grant Account Users Read Access to Volume

• Click on the bp-lakehouse-monitor volume in ‘Catalog Explorer’

• Select Permissions

• Click on Grant

• Select READ VOLUME and add ‘Account Users’ principal

• Click on Grant

Unity Catalog Disabled

This configuration step is done automatically by LHO when LHO Telemetry Agent is updated.

LHO copies the cluster-init-script into the storage area accessible by the cluster, i.e. the Databricks workspace files

/Workspace/bp-lakehouse-monitor/script/init_script.sh

Related articles

• Auto-Pilot Monitoring (AWS)