Below is a checklist designed to get you ready to deploy the Lakehouse Optimizer (LHO) on AWS
- Decide which Databricks workspaces should be monitored.
If all your workspaces are in the same AWS account and you plan to deploy LHO into that same account, you will be deploying the Single AWS account scenario.
Otherwise, you are deploying the Cross-account AWS scenario.
- Determine which AWS region the LHO infrastructure will be deployed into. Ideally the same region as your Databricks workspaces
- For your selected region, insure there’s an available VPC slot as the installation create a new VPC.
- Understand AWS infrastructure resource requirements: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2577662006/AWS+Resource+Requirements#Required-AWS-Resources
- The IAM user account running the installation needs the following policies attached: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2664071177/Deployment+and+Quick+Setup+Guide+AWS#AWS-Account-Prerequisites
- If you have Microsoft Entra ID (Azure Active Directory) as your identity provider, create an app registration in your tenant and https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2664071177/Deployment+and+Quick+Setup+Guide+AWS#Azure-Active-Directory-Single-Sign-On-Prerequisites
- Have the following values available:
- App registration client ID
- Azure AD tenant ID
- The client secret value
- Create the Databricks service principal – https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2577662006/AWS+Resource+Requirements#Databricks-Service-Principal
- Account Admin
- Workspace Admin for all planned monitor workspaces
- Determine identity authorization solution for the LHO agent running on Databricks Workspace compute resources. Options outlined here -- AWS Resource Requirements - Blueprint Lakehouse Optimizer Documentation - Confluence (atlassian.net)
- Depending on options taken:
- Create IAM user, saving username, access key and secret for later use.
- Gather all instance profile ARNs in use by all target workspaces for later use, or have root account ARN available
- Expand all in-use instance profile IAM policies to include the permissions required for the LHO Agent.
- Enable CostExplorer in AWS https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2577662006/AWS+Resource+Requirements#Tags-to-activate-in-Cost-Manager
- Sometime after the deployment completes, activate the documented user-defined tags
Add Comment