If a signed-in Lakehouse Monitor (LHM) user does not have access to any Azure Subscription and/or Databricks Workspace, but the LHM admin still wants to provide read rights to that particular user (or group), the admin can use publicSubscriptionMetadata LHM config file to make subscriptions and workspaces accessible in LHM. This is an application-wide setting. Use publicSubscriptionMetadata.csv file to provide read-only access to Subscriptions and Workspaces to any user that is able to login to LHM App.

Private & Public metadata files

Lakehouse Monitor (LHM) keeps public and private metadata files which contain Subscriptions and Databricks workspaces information (i.e. IDs).

Private file is used only by consumption library to load consumption data only for these workspaces. Workspaces are only added to the file by using user privileges, service principal privileges and workspaces from the public file (see more details below).

• subscriptionMetadata.csv

Public file is used in order to list workspaces. This file must be manually generated in storage account under a folder with the name as the subscription id, in a file named publicSubscriptionMetadata.csv. Public file is only read by BPLM app.

This file is used to allow signed-in users to list workspaces and see reports in LHM even if that particular user does not have a user configured the tenant that manages the subscription for Databricks.

For example, the user signs in with an external account registered with a tenant from another Microsoft AD subscription, thus when a signed-in user doesn’t have the sufficient rights to read workspaces from the subscription that also manages the Databricks workspaces. In order to allow this user to read workspaces that the admin deemed “public”, LHM uses the publicSubscriptionMetadata.csv to expose these workspaces to this external user.

Manual Public Subscriptions and Workspaces Configuration

Step 1 – Define Public Subscriptions manually in cloud storage

• use cloud storage path from LHM Settings page

  • 

• define in blob storage a file named

publicSubscriptions.csv

• one row for each subscription

• example:

displayName,subscriptionId,tenantId
Blueprint Data Engine,a63c1e51-40ae-4a34-b230-bf80e132c05c,12e2dd65-5024-44c2-83b5-3ca21c04ef0e

Step 2 – Create a folder with subscription-id as name in cloud storage

In this folder, we have two files

• (1)publicSubscriptionMetadata.csv

  • create the file manually

  • add metadata for workspaces that you want to consider “public”

  • example:

  • displayName,isPremium,workspaceHost,workspaceId,workspaceResourceGroupId,workspaceResourceId
    ca-adb-test-workspace,false,adb-511420607229897.17.azuredatabricks.net,511420607229897,/subscriptions/a63c1e51-40ae-4a34-b230-bf80e132c05c/resourceGroups/databricks-rg-ca-adb-test-workspace-loj4oc72jjfum,/subscriptions/a63c1e51-40ae-4a34-b230-bf80e132c05c/resourceGroups/cost-analyzer-resources/providers/Microsoft.Databricks/workspaces/ca-adb-test-workspace

• (2) subscriptionMetadata.csv

  • created automatically by LHM App

  • generated when we create or update a schedule or when we run on demand consumption loading.

    • see LHM Consumption Management page

Metadata Files containing

subscriptionMetadata.csv

• specifies for which workspaces from a given subscription we load consumption data

• update the private file every time user click on run now or save/update scheduler , in my opinion our consumption loader should only read private file. That file should always contain up to date list with workspaces.

• file is updated when consumption loading is run

  • file updated when we create or update a schedule or when we run on demand consumption loading.

• the file will be updated with the following workspaces list:

  • (1) workspaces seen by using rights provided by managed identity or service account

  • (2) workspaces seen by using rights provided by signed-in user

  • (3) workspaces already configured in publicSubscriptionMetadata.csv

• concatenated list of workspaces from (1) ++ (2) ++ (3) is saved in subscriptionMetadata.csv

• config: AUTO_UPDATE_PRIVATE_METADATA_FILE

  • use flag if we should update metadata file or not

  • AUTO_UPDATE_PRIVATE_METADATA_FILE -> default True.

  • if true – file is updated automatically

  • if false – file is updated manually

  • file is always used by Consumption Management module regardless if this config is true or false