This guide is meant to guide you through the Lakehouse Optimizer install process using Azure Marketplace. It is designed to get at least one of your workspaces fully configured to the point where you are seeing reporting in Lakehouse Optimizer (LHO) on your Databricks entities like Jobs, Workflows, All Purpose Clusters, Pools, and more. We will also help you set up the parent subscription to load consumption data from Databricks allowing for cost analysis data to be shown in reports and overview graphs and charts.
...
Requirements
To ensure you have all the permissions required to do everything in this guide, please check out our Azure Marketplace Installation Readiness Checklist.
Azure Marketplace Installation
Go to the Lakehouse Optimizer for Azure Databricks page in Azure Marketplace. Select Free Trial and click the Create button.
...
Again, Application, Virtual Machine, and Storage Account names must all be unique, though you can make this easy by adding a suffix onto the end of the resource group name you picked on the last page as shown above.
Admin Email Address
NOTE: Be sure to use a valid email address in the “Admin Email Address” field because later on when the SSL certificate eventually expires, this email is used to receive the notification email from letsencrypt
The email address and password will be the login credentials after installation, so be sure to write them down somewhere as there will not be a way to retrieve the provisional user username/password if you forget them.
SSH Public Key
The SSH key is a generated access key that will allow you to directly access the virtual machine generated by this installation and make customizations to the resources. Note that SSH access is only granted to connections that are coming from Azure CloudShell, so trying to connect using your local PC terminal won’t work even if you have the correct SSH key.
...
The installation process should take about 10 minutes to complete.
Login as Provisioning User
...
Once installation is complete you can click Go to resource to open the managed application resource, and then click the managed resource group link in the upper right hand side of the page to view all the installed Azure resources.
...
Tenant Id - the Directory (tenent) ID (can be copied easily in App Registration
Client Id - the Application (client) ID from the App Registration
Secret - The Value of a client secret in your App Registration
Object Id - Object Id of the managed application in the App Registration
Create an App Registration
Azure Marketplace does not allow the creation of this specific resource during the install process due to security reasons, so we will create an app registration now.
...
https://[your-dns-name-url]/login/oauth2/code/azure
Click Register.
Implicit grant and hybrid flows
Next, we need to grant LHO access to subscription data. Go to Authentication on the left hand side nav, and scroll down till you see the section “Implicit grant and hybrid flows”. Check the box next to ID tokens (used for implicit and hybrid flows).
...
Client Secret creation
Go to Certificates and Secrets on the left hand side navigation, and click New client secret. Give it a name (doesn’t have to be unique), and pick a time period for how long till it expires.
...
Be sure to copy the Value field from the client secret as you will need it in the next section, and after you leave this page you will no longer be able to access the full value.
Enter the info in LHO
...
Enter Secret
The secret you need is the Value from the client secret you created in the section above. Paste that value in the Secret field in LHO.
...
Enter Tenant Id
...
This value is found in the App Registration overview.
...
Copy each value into their respective fields in LHO Directory (tenant) ID in the respective LHO field.
Enter Object Id and Applicaiton (Client) ID
To get the object id first click on the link for the managed ‘Managed application inside in your app registrationlocal directory’. You will find the link on the right and side of the overview page for the App Registration.
...
Once on that page, you should see the Object ID and Application ID in the overview header section.
...
Copy the value values into the LHO field and hit save. Once everything applies you should see a big green check mark that says “Configured”.
...
Log in using Active Directory SSO
Now that Active Directory is configured you can log out, and you should now see the option to sign in using Single Sign On (SSO).
...
Click Accept and you will be taken to the overview page, but there will be no data yet because no workspaces are configured.
...
Configure Workspaces and Subscriptions in LHO
Configure Azure Subscription
Step 1. Grant Access to Consumption Data
...
You can read more about access configuration here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2532605979/Security+Requirements+for+VM+runtime#Phase-3)-Access-roles-configuration
Configure Databricks Workspace
The following actions are required in order to enable Lakehouse Optimizer to gather cost and telemetry data:
...
This setup is the quickest option to get your Databricks monitored. There are also other configuration options for LHO, for example to enable monitoring on assets one-by-on. For more configuration options please contact Blueprint or follow the more advanced topics in the documentation material.
Load Consumption Data
Step 1. Navigate to the Consumption Data panel.
...
You can configure multiple schedules based on your particular needs.
...
VI. Explore Cost and Telemetry Insights
Once all previous steps are completed, your LHO instance is ready to monitor your cloud infrastructure.
...