Installation Prerequisites

Helpful Competencies During Deployment

To aid in following the below steps and understanding the installation procedure, knowledge about AWS services such as EC2, VPC, RDS, Secrets Manager, and IAM Roles\Policies. Additionally, Linux administration, bash scripting, and optionally Azure Entra ID app registrations is suggested but not strictly required.

AWS Account Prerequisites

The AWS account (the “deployer” AWS IAM user) used to run the LHO installation script must have the following rights already granted in order for the installation process to complete successfully. Below is the AWS IAM policy in JSON format.

Databricks metastore admin permission

Besides the AWS permissions listed above, the deploying user needs to be a Metastore Admin of the Databricks Unity Catalog. We recommend creating a group which shall be configured as Metastore Admin and having the admins added to this group.

Supported AWS Regions

The Lakehouse Optimizer supports any region where all the required AWS services can be deployed. You can check service availability by region by this AWS page to list AWS services available by region.

Databricks Service Principal

Ensure the Databricks service principal was created with the prerequisites linked below

AWS Resource Requirements | Databricks Service Principal

LHO Agent Access

Make a decision on how the LHO agent will be granted authorization in your Databricks environment. We suggest creating an IAM user and configuring a role trust, as outlined in the documentation linked below.

AWS Resource Requirements | AWS Permission Policies

Single Sign-On Configuration

Lakehouse Optimizer supports a handful of different identity providers. Please follow the links to the configuration page for your SSO provider of choice. If you use Databricks accounts for authentication, you can skip this section.

Microsoft Entra\Azure Active Directory

Okta

Google

I. Installation Guide

Step 1. Log in to AWS and open AWS CloudShell

Open

Step 2. Prepare Lakehouse Optimizer deployment archive.

Step 3. Build the argument list and run deployment script.

There are multiple options for LHO for AWS deployment, please understand the below options and create an argument list that best configures LHO for your environment’s needs.

The base arguments for all deployments are below:

Single Sign-On Options

Azure Entra Id ( Active Directory )

Two additional arguments must be passed in, ‘tenant_id' and 'service_principal'. You will also be prompted for the client secret during script execution

Okta

Okta SSO requires that the Okta base URL and client ID be added as arguments. You will also be prompted for the client secret during script execution

Google

Google SSO requires the Google app client ID passed in as an argument. You will also be prompted for the client secret during script execution

LHO for smaller environments

If you are deploying a smaller footprint version of LHO, you can leverage the switch --docker_hosted_sql. This will deploy SQL Server container hosted on the same host VM as the LHO app container. This switch can be used in conjunction with any combination of the Azure AD enabling arguments as well as the below arguments for configuring the LHO agent.

Options for LHO Agent authorization

Documentation for determining what authorization solutions works best, see: AWS Resource Requirements | Lakehouse Optimizer Agent Permissions

Configuring option 1, leveraging an IAM user for LHO agent access

After creating an IAM user and assigning the LHO agent policy (outlined in documentation link above):

--iam_agent_username - Username of AWS IAM user created and assigned the LHO Agent policy to access DynamoDB and SQS.

Configuring option 2, using role trusts

The below code snippet creates a role trust file that is a newline separated list of all the instance profile role ARNs or AWS accounts that are in use in your Databricks environment. This file location is passed in as an argument to the deployment script.

Copy the snippet to a text editor. Change the below ARN values defined in the role_array variable declaration. After updating, run in AWS Cloudshell to create the role trust file.

Your argument list:

--role_trusts_file The location of the created role trust list file. If following the above steps to create the file, the location would be ~/lho/agent_trusted_roles.txt

Argument recap

There are multiple different argument options available to configure the LHO application to suit your needs. Beyond the required base arguments, you can mix and match the additional options discussed above. For example, you can run an Azure AD SSO version of LHO that enables both IAM user and role trust authentication solutions. For other options, run ./deploy-lm.sh --help to see all available options

Step 4. Script input fields:

 Result of process is:

Step 5. Installation complete

Once the installation is complete, you will see the following output.

The URL to login to LHO will be printed in the Cloudshell output.

Please copy the App URL that you will use to login to LHM.

if you got link with http protocol - change it to https

Step 6. SSH login (optional)

Once the script is done you also have the ssh key to access the VM in your ~/.ssh folder in the AWS cloudshell session.

You can use this to login to the host VM and gather logs from the container via docker logs

docker logs -n 500 -f

II. First-Time Login Guide

Step 1. Login to LHO App

with the login URL provided when the installation was complete.

Open

Step 2. Grant permissions

If it’s the first time you are logging in with your user to LHO, you will be asked for permissions by LHO’s App Service. Click Accept.

Approval Required Troubleshooting

Depending on how your Azure subscription is configured by the IT department, you might also come across the following “Approval required” screen. The approval must be granted by the IT department. It is not something related to configurations done by LHO installation process.

Please follow the following guide to avoid this dialog:

Active Directory Enable Access for All Users at Tenant Level

If you still have issues sign-in in and thee see the screens below, please follow the next steps:

Step 3. Add app role

Open created Azure App → App roles → Create app role with value bplm-admin

Step 4. Assign user and group

Open created Azure App overview page → Managed application in local directory → Assign users and groups → Add user/group → add a new role bplm-admin → Select your user and save

Step 5. Add the redirect uri to the registered app

Open registered app Overview page on Azure Portal

Open Redirect URIs page

Click on Add platform

Choose Web and insert new redirect URI

Step 6. Configure License

Once logged in, you will be redirected to the License panel.
Need to add new Assignment for bplm-admin

Copy the License Token and contact Blueprint and provide the token in order to receive a trial or permanent license for your deployment.

Once you receive the license, add the License Key and Public Key in this panel.

Once this is done, LHM is ready to start monitoring your assets.

III. Configure Databricks Workspace

The following actions are required in order to enable Lakehouse Monitor to gather cost and telemetry data:

1. Go to → Settings → Provisioning & Permissions

2. Generate your Databricks token

3. Add token

This setup is the quickest option to get your Databricks monitored. There are also other configuration options for LHM, for example to enable monitoring on assets one-by-on. For more configuration options please contact Blueprint or follow the more advanced topics in the documentation material.

IV. Load Consumption Data

Step 1. Navigate to the Consumption Data panel.

This page is available only to the role of Billing Admin.

Step 2. Load Now consumption data

LHO supports loading consumption (cost/usage detail) data from both your Databricks account for your DBU charges as well as AWS Cost Explorer for your AWS cloud chages, either on demand or on a schedule basis.

At this step, for this tutorial purpose, select Run Now and load data for the past 30 days or 2 months at most. Depending on your Azure Subscription size this process might be long, therefore we recommend to load for a smaller date interval, the purpose being to see cost and telemetry data in LHO as soon as possible.

Loading consumption data for large subscriptions for the past 12 months, can take up to 12 hours or even more.

Step 3. Scheduled load consumption data

Most likely, Databricks resources are used on a daily basis in your infrastructure. Therefore we recommend you to create a scheduled daily consumption data load in order for LHO to report updated costs on a daily basis.

Recommended schedule configuration:

• load data: incrementally

• frequency: daily

You can configure multiple schedules based on your particular needs.

V. Explore Cost and Telemetry Insights

Once all previous steps are completed, your LHO instance is ready to monitor your cloud infrastructure.

VI. Automatically grant access consent for all Active Directory Users (optional)

The following guide will help you configure the login process such that users with a valid AD user using single-sign-on will login automatically, without having to click on “grant permissions” dialogs or contact IT for further approvals.

Active Directory Enable Access for All Users at Tenant Level

VII. Assign User Roles in Lakehouse Optimizer (optional)

If Azure Active Directory is used for authentication, then each user can also be assigned to different roles supported by Lakehouse Optimizer.

The following article provides further configuration details:

What roles are there in the LHO app?

• Lakehouse Optimizer User Roles

How can I assign LHO roles to users?

• Lakehouse Optimizer User Roles

Related

• AWS Resource Requirements

• AWS Deployment Diagram

• Guides and Tutorials