Azure Account Prerequisites

The Azure account used to run the LHO installation script must have the following rights already granted in order for the installation process to complete successfully.

1. Resource Group Owner

• the user must have the ability to create a resource group or be assigned as owner of the resource group in which LHO resources will be installed

2. Application Developer

• the user must be assigned Application Developer role in order to be able to create LHO application’s service principal

Open

application developer role

3. UserAccessAdministrator Role

The signed in user will grant the application the necessary permissions to load consumption data on a schedule and analyze telemetry data. The signed in user must have at least the UserAccessAdministrator role in the subscription.

4. Databricks Metastore Admin

The user configuring the Optimizer the first time will need to be a Metastore Admin inside of the Databricks Unity Catalog. We recommend creating a group and assign it as the Metastore Admin, add admins as members to this group.

5. Databricks CREATE_VOLUME for main catalog

The user configuring the Optimizer the first time will need to be have the CREATE_VOLUME permission on the main catalog. Requirements no 4 (above) and 5 are needed so that the Lakehouse Optimizer init script get’s uploaded to the Databricks Unity Catalog and the Catalog configured to use it.

I. Installation Guide

Step 1. In the Azure portal create a resource group for your deployment

Open

resource group creation

If you don’t have sufficient rights to create a resource group, you will need to have contributor role assigned for this resource group to be able to run the installation script.

Step 2. Open up a PowerShell in the Azure portal.

Use PowerShell prompt for next steps.

If you are starting PowerShell for the first time, you need to add storage permissions for your account.

Storage Account contributor on the Azure Subscription will grant you sufficient rights to start PowerShell.

Step 3. Run the below code snippet to download the deployment archive, unzip it, and change directory into the newly expanded archive:

Unzip using:

Step 4. Prepare deployment information by gathering some information and creating the parameter object

• subscription id

  • you can find this ID in Azure Portal in the view with the resource group you created

• resource group name

  • you can find this ID in Azure Portal in the view with the resource group you created

• base application name

  • used by Azure App Registration service to name

  • represents the name of the App Registration of LHO

  • it can be any name that you will use for LHO deployment

    • for a guideline on naming convention please see: Define your naming convention - Cloud Adoption Framework

  • note that this will be used to name the Azure KeyVault and Storage Accounts. The KV name must be globally unique (across the entire Azure space) so we recommend using specific names instead of generic ones. E.g: lho-<your company name here> instead of lho

  • Naming restrictions apply for storage accounts. The script will automatically remove any '-' characters

• DNSPrefix

  • A descriptive short name that is registered with Azure’d internal DNS and will be used in the application URL. Should describe the app’s usage. eg: lho-companyshortname-dev

  • Naming restrictions apply for DNS names: valid characters a-z,0-9,'-'

• docker container registry username and password

  • contact Blueprint for these credentials

  • input field ACRUsername

Step 5. Run the script ./vmdeploy.ps1 @params
This script creates all the resources required by LHO. See more here:

• Azure Resource Requirements | Required Azure Resources

You will be prompted to enter the ‘ACRPassword' that corresponds to the username provided above. There will also be an SSH key generated and a password to input for that key if desired. The private key will exist on the user’s cloudshell ~/.ssh/ directory.

You are expected to see similar information as in the screenshots below during the installation process:

Open

Some steps might take quite a few minutes to create. For example, a complete installation is expected to take around ~10 minutes.

Step 6. Installation complete

Once the installation is complete, you will see the following output.

The URL to login to LHO will be printed in the PowerShell output.

Please copy the App URL that you will use to login to LHO.

Step 7. SSH login (optional)

Once the script is done you can use the ssh key you generated to access the VM in the PowerShell session.

You can ssh to the VM with the following command:

For example:

Troubleshooting Errors

The following error message is caused by Insufficient Permissions on the user that is running the installation script.

Failed [ManagedIdentityCredential authentication failed: Service request failed. Status: 400 (Bad Request)

How to check if you have sufficient permissions?

Get-AzADApplication

Run this command to check that you have enough permissions to list applications with Active Directory.

The above command might fail also because of integration issues between Azure services. Even with correct permissions, the above command might fail with a 400 error.

Getting inner details

With the infrastructure deployment failures, the displayed error message in cloudshell is not always helpful. You can get the inner details by finding the tracking id GUID in the error message and running the below cmdlet:

II. First Login Guide

For Unity Catalog enabled workspaces

If one or more Databricks workspaces you intend to monitor with LHO have Unity Catalog enabled, there is extra configuration required to upload and whitelist the LHO agent init script to a shared volume. Please follow the steps outlined in the link provided before continuing with the first time login

Provisioning with Unity Catalog Enabled (AWS) - Blueprint Lakehouse Optimizer Documentation - Confluence (atlassian.net)

Assign workspace read permissions via Azure AD custom role

Listing workspaces in each available subscription requires read permission for the signed in user as well as the LHO Service Principal. If no workspace is listed, then this permission is not already assigned. The Microsoft.Databricks/workspaces/read permission can be granted via a custom role at either Azure subscription or resource group level containing the Databricks workspaces this user should be able to access from the application.

Read more about it here: Azure Security Requirements for VM runtime | Phase 2) Azure AD SSO user requirements

Step 1. Login to LHO App

with the login URL provided when the installation was complete.

Step 2. Grant permissions

If it’s the first time you are logging in with your user to LHO, you will be asked for permissions by LHO’s App Service. Click Accept.

Approval Required Troubleshooting

Depending on how your Azure subscription is configured by the IT department, you might also come across the following “Approval required” screen. The approval must be granted by the IT department. It is not something related to configurations done by LHO installation process.

The following guide will help you configure the login process such that users with a valid AD user using single-sign-on will login automatically, without having to click on “grant permissions” dialogs or contact IT for further approvals.

Active Directory Enable Access for All Users at Tenant Level

Step 3. Configure License

Once logged in, you will be redirected to the License panel.

Copy the License Token and contact Blueprint and provide the token in order to receive a trial or permanent license for your deployment.

Once you receive the license, add the License Key and Public Key in this panel.

Once this is done, LHO is ready to start monitoring your assets.

III. Configure Azure Subscription

Step 1. Grant Access to Consumption Data

Navigate to Settings panel and grant access to the consumption (cost) data to the Service Principal used by LHO.

In order for Lakehouse Optimizer (LHO) to be able to read consumption data from Azure, LHO's application identity requires the BILLING_READER role to be granted in this Azure subscription.

Once this step is complete, you will see the following green check mark.

LHO can function also without consumption (cost) data access, but this means that LHO will not be able to report on your actual costs.

You can read more about access configuration here: Azure Security Requirements for VM runtime | Phase 3) Access roles configuration

IV. Configure Databricks Workspace

The following actions are required in order to enable Lakehouse Optimizer to gather cost and telemetry data:

• Grant Access to Service Principal

• Enable LHO Collector Agent

• Enable Global Init Scripts

Step 1. Enable Service Principal

• The service principal configured for this application to do single-sign-on is also used for analysis processing by the use of external native Databricks notebooks.

• Service Principal was configured automatically by the installation scripts

• You can read more about Service Principal configuration here:

  • Azure VM/Docker with Azure Service Management and Service Principal | How do I configure the Azure Active Directory Service Principal?

  • Azure VM/Docker with Azure Service Management and Service Principal | How is Databricks managed in LHO? (4)

Step 2. Enable LHO Collector Agent

• Upload .jar library responsible for collecting telemetry data and the initialization scripts into selected workspace DBFS.

Step 3. Enable Global Init Scripts

Step 4. Create Secret Scope

Step 5. Configuration Complete Confirmation

Once these steps are done, you should see the following green banner with “Complete Configuration”.

This setup is the quickest option to get your Databricks monitored. There are also other configuration options for LHO, for example to enable monitoring on assets one-by-on. For more configuration options please contact Blueprint or follow the more advanced topics in the documentation material.

V. Load Consumption Data

Step 1. Navigate to the Consumption Data panel.

This page is available only to the role of Billing Admin.

Step 2. Load Now consumption data

LHO supports loading consumption (cost) data from your Azure subscription either on demand or on a schedule basis.

At this step, for this tutorial purpose, select Run Now and load data for the past 30 days or 2 months at most. Depending on your Azure Subscription size this process might be long, therefore we recommend to load for a smaller date interval, the purpose being to see cost and telemetry data in LHO as soon as possible.

Loading consumption data for large subscriptions for the past 12 months, can take up to 12 hours or even more.

Step 3. Scheduled load consumption data

Most likely, Databricks resources are used on a daily basis in your infrastructure. Therefore we recommend you to create a scheduled daily consumption data load in order for LHO to report updated costs on a daily basis.

Recommended schedule configuration:

• load data: incrementally

• frequency: daily

You can configure multiple schedules based on your particular needs.

VI. Explore Cost and Telemetry Insights

Once all previous steps are completed, your LHO instance is ready to monitor your cloud infrastructure.

Select Reports and select the Azure Subscription and Databricks Workspace you just configured.

VII. Automatically grant access consent for all Active Directory Users (optional)

The following guide will help you configure the login process such that users with a valid AD user using single-sign-on will login automatically, without having to click on “grant permissions” dialogs or contact IT for further approvals.

Active Directory Enable Access for All Users at Tenant Level

VIII. Assign User Roles in Lakehouse Optimizer (optional)

If Azure Active Directory is used for authentication, then each user can also be assigned to different roles supported by Lakehouse Optimizer.

The following article provides further configuration details:

What roles are there in the LHO app?

• Lakehouse Optimizer User Roles

How can I assign LHO roles to users?

• Lakehouse Optimizer User Roles

Related Content

• Azure VM/Docker with Azure Service Management and Service Principal

• User Authentication Flow Diagram with Active Directory in Azure

• Guides and Tutorials