Initial Setup and Configuration (Azure)
This quick setup guide outlines how to enable cost and telemetry monitoring on LHO, starting from the first login after a successful deployment.
Table of contents:
- 1 Permissions required during the first LHO configuration:
- 2 First Login Guide
- 3 Automatically grant access consent for all Active Directory Users
- 4 Assign User Roles in the Lakehouse Optimizer
- 5 Set License
- 6 Configure Azure Subscription
- 7 Provision Databricks Workspace
- 8 Load Consumption Data
- 9 Where to go from here?
Permissions required during the first LHO configuration:
The signed in user must have at least the UserAccessAdministrator role in the subscription.
The user configuring the LHO the first time will need to be a Metastore Admin inside of the Databricks Unity Catalog. We recommend creating a group and assign it as the Metastore Admin, add admins as members to this group.
The user configuring the LHO the first time will need to have the CREATE_VOLUME permission on the main catalog.
First Login Guide
For Unity Catalog enabled workspaces
If one or more Databricks workspaces you intend to monitor with LHO have Unity Catalog enabled, there is extra configuration required to upload and whitelist the LHO agent init script to a shared volume. Please follow the steps outlined in the link provided before continuing with the first time login - Provisioning with Unity Catalog Enabled
Assign workspace read permissions via Azure AD custom role
Listing workspaces in each available subscription requires a custom role with a special permission assigned to the LHO Service Principal. The Microsoft.Databricks/workspaces/read permission can be granted via a custom role at either Azure subscription or resource group level containing the Databricks workspaces
Important: Follow the steps described in this section to create the custom role at the Subscription level
In your Azure Portal go to Subscriptions
Select the subscription and go to Access Control (IAM)
Click on the + Add button and select Add custom role
Give your role a name, e.g: DatabricksRead
Under Baseline permissions choose Start from scratch and click Next
Click on Add Permissions
Search for databricks and select Microsoft Databricks
Under Microsoft.Databricks/workspaces check the box for Read: List Databricks Workspaces and click Add the permission should be in the list
Click Next, the scope should be the subscription (if you chose to add the role at a resource group level the scope should be the specific resource group that contains the Azure Databricks resource)
Click Next, the JSON should look like this
{ "properties": { "roleName": "DatabricksRead", "description": "", "assignableScopes": [ "/subscriptions/<your subscription ID>" ], "permissions": [ { "actions": [ "Microsoft.Databricks/workspaces/read" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } }Create the role and assign it to LHO Service Principal
Repeat the steps for all subscriptions in your account
Step 1. Login to LHO App
with the login URL provided when the installation was complete.
Step 2. Grant permissions
If it’s the first time you are logging in with your user to LHO, you will be asked for permissions by LHO’s App Service. Click Accept.
Automatically grant access consent for all Active Directory Users
Configure the login process so that users with a valid Active Directory (AD) account can automatically log in using single sign-on (SSO), without needing to click “Grant Permissions” dialogs or contact IT for additional approvals. Follow this guide:
Assign User Roles in the Lakehouse Optimizer
When Azure Active Directory is used for authentication, each user can be assigned a specific role supported by the Lakehouse Optimizer.
It is essential to define roles in the Service Principal app to restrict default access permissions to the read-only User LHO role, with viewing rights governed by the permissions configured in Databricks.
Follow this guide to configure LHO user roles:
Set License
Go back to LHO where you logged in, you will be redirected to the License page.
Copy the License Token and provide the token to the Blueprint team in order to receive a trial or permanent license for your deployment.
The Blueprint team will provide you with a License Key and Public Key via email.
Once you receive the email:
Add License Key
Add Public Key
Click Apply & Reload
Configure Azure Subscription
This step allows LHO to report on your actual costs.
In order for Lakehouse Optimizer (LHO) to be able to read consumption (cost) data from Azure, LHO's application identity requires the BILLING_READER role to be granted in this Azure subscription.
User configuring this must have User Access Administrator rights in Azure Subscription and the Admin role in LHO.
Navigate to Settings > Provisioning & Permissions page.
Select Azure Subscription for which consumption data should be loaded
Click Grant for Service Principal button.
The button is not enabled if the user doesn’t have permission to grant the BILLING_READER role in selected Azure Subscription.Verify that a green check mark appears after clicking the button.
Repeat step 2-4 for all Azure Subscriptions that should have consumption data loaded.
Provision Databricks Workspace
Follow Workspace Provisioning in LHO (Azure) guide to complete this section and enable workspace telemetry monitoring.
Load Consumption Data
Follow Initial Consumption Data Load and Configuration guide to complete this section.
Where to go from here?
Once all previous steps are completed, your LHO instance is ready to monitor your cloud infrastructure. Refer to LHO feature guides how to explore cost and telemetry insights: