Reference links

• Azure Service Management

• Azure Service Principal

• What are managed identities for Azure resources?

• Azure Key Vault

• Databricks Secrets

See networking diagram here:

https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/3596648450

Signing into the Lakehouse Optimizer application URL is done via Azure Active Directory credentials that are authorized for access into all available subscriptions in the Azure tenant and for all available workspaces in the available subscriptions.

• LHO deployment scripts create and configure an App Registration in your Azure portal Microsoft Azure portal App Registrations for Azure AD Single Sign-On and as the application identity for calling downstream Databricks APIs for background telemetry data analysis

• LHO requires an administrator rights when running deployment scripts.

• Configurations done automatically by the deployment scripts:

  • Creates an Azure AD App Registration that will be used as a Service Principal for Azure AD Single Sign-On

  • Creates an App Registration in your Azure portal Microsoft Azure portal App Registrations

    • sets a name for the Service Principal. This name will be used later to assign roles

  • Sets the redirect uri to https://{FQDN}/login/oauth2/code/azure where FQDN is the url the LHO Application is published with

  • Creates a secret (Certificates & Secrets tab) named msft-provider-auth-secret , also known as client secret

    • in Azure Key Vault sets the LHO secret msft-provider-auth-secret to <value-of-msft-provider-auth-secret>. The Azure Key Vault instance was already created by the LHO deployment script with the name specified during deployment process.

  • Enables ID Tokens in the Authentication tab

  • sets clientId, tenantId as public variables for LHO .env file (3)

• you can find the created Service Principal by searching in Microsoft Azure portal App Registrations with the clientId exposed in the LHO Workspace Settings

Open

Service Principal identification

Open

LHO App Registration

By using Azure Service Principal

The user who installed the Lakehouse Optimizer application is assumed to already have User Access Administrator role. This role is required to install LHO app using the deployment scripts. See steps of “How do I configure the Azure Active Directory group?”.

Billing Reader role is required to be granted to the Service Principal for loading consumption data for a target Azure subscription.

The LHO Application Settings page provides a button for the grant. This function however requires the Azure Service Management Functions to be enabled, which is the case in this configuration.

Therefore, the following automated procedure will be executed.

Open

The administrator configuring LHO application can grant the necessary permissions to load consumption data directly from the Setting panel of LHO app - click on “Grant Access to Service Principal” button. The signed in LHO user must have at least the UserAccessAdministrator role in the subscription to properly configure the necessary permission.

The following configuration steps are done automatically by LHO:

• in Azure Portal → Subscriptions → Azure subscription → Access control (IAM) → Role assignment

• Billing Reader role is added to Members of the Service Principal identity

  • see more details at “How do I configure the Azure Active Directory group?”

Billing Reader role at Azure subscription level is required in order to allow the LHO application to read consumption/usage details data.

Any signed in Azure AD user that has the admin role in the LHO application can access the Consumption Management page in LHO and trigger the loading of consumption data from Azure to LHO.

The consumption data is loaded by using the Service Principal of LHO app configured previously, see step “How can I grant rights to LHO app to read consumption (cost) data?”. The signed in user does not need any additional rights assigned to trigger this operation.

• Service Principal Client Secret Secret

  • see “How do I configure the Azure Active Directory Service Principal?”

• LHO Azure SQL/SQL Server database user & password

  • see “LHO Installation Manual” (contact Blueprint)

  • saved in Azure Key Vault

This LHO deployment configuration uses Azure Key Vault to store and retrieve secrets.

Read more here: Azure Key Vault.

• LHO app can read all environment variables configured the virtual machine environment

• Service Principal Client Secret

  • see “How do I configure the Azure Active Directory Service Principal?”

  • used by the LHO App for SSO via OAuth2 Web Flow, using the configured Service Principal

  • used by Service Principal to access Azure Blob Storage, Containers and Queues on behalf of the designated Service Principal

  • used by the LHO App to call the Databricks API on behalf of the designated Service Principal

• LHO database user & password

  • see “LHO Installation Manual” (contact Blueprint)

  • used by LHO app to read and write data to LHO database deployed as Azure SQL Server (10)

  • saved in Azure Key Vault

• A Databricks user must be created in Databricks (see Workspace/Settings/Admin Console/Users) for each user that signs into LHO via Azure AD. The LHO app does not create users automatically as it would require a higher privilege for the signed in user.

  • The Databricks user’s permission will dictate the level of access to entities and resources. If the workspace is part of the Premium tier, more granular access control can be enabled, and the “Can Manage” permission is required for this user to change cluster or job configuration.

  • For non-Premium workspaces, any Databricks user can edit configurations, the one exception is All Purpose clusters, they can be only edited by their Owners, since we introduced Databricks secrets in configuration.

• Authorization: Lakehouse Optimizer calls Databricks APIs on behalf of the signed in Azure AD user or Service Principal, depending on context, so authorization is performed by Databricks.

• Authentication for the downstream Databricks API calls is done via OAuth2 Access Tokens (OnBehalfOf flow) https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-on-behalf-of-flow

  • For more details see https://blueprintconsultingservices.atlassian.net/wiki/spaces/BLMPD/pages/2547286017

• LHO App Settings page provides a feature called “Add Service Principal” that will create a Databricks user for the configured Service Principal in the selected Databricks workspace.

  • It uses the SCIM Databricks API under the hood.

• Any LHO Azure AD signed in user will be authorized by Databricks through the API calls made by the application on the user’s behalf. See the full list of permissions required for a user to have access to all the features in LHO.

• Any LHO logged in user will see Databricks entities based on the rights configured for that user in Databricks. LHO uses the signed-in user AD access token to list Databricks entities.

• LHO app will use the Service Principal to call Databricks APIs for telemetry analysis.

  • to create a Service Principal to use for Databricks please see “How is Databricks access managed in LHO?”

• As a background process, LHO app will use the Service Principal user to read Databricks entities.

  • to create a Service Principal to use for Databricks please see “What rights does LHO need to access Databricks entities?”

  • Managed Identiy used by LHO is created at step “How do I configure the Azure Active Directory group?”

• The Service Principal represents the application identity is it will require access to all the Databricks entities the LHO App monitors: clusters, jobs, DLT pipelines, SQL Warehouses. Analysis data will be stored in the database, used in reporting. The Signed in Azure AD user will be authorized at runtime by the downstream API calls and the application will filter the content of the reports generated from the analyzed database data with the authorization rules in Databricks. Any signed in user will only be able to access the Databricks entities is authorized for, even though the database will have analyzed data for all (via Service Principal)

• The LHO “Add Service Principal” feature on the Settings page will add the SP Databricks user to the admins group. The exact list of required permissions however is documented below:

• In order to monitor an entity (e.g. workflow, cluster, pipeline), LHO needs to update the configuration of that entity. Databricks allows only some of the users to update modify the configuration of an entity (e.g. administrators or owners of clusters).

• LHO uses the rights of the signed in user to enable monitoring of an entity.

  • If the signed in user can modify the configuration of an entity in Databricks, then LHO also will be able to monitor that entity.

  • LHO does not use the Service Principal rights to enable or disable monitoring of an entity.

• If the signed in user can modify the configuration of an entity in Databricks, then LHO also will be able to monitor that entity. Collecting telemetry data does not require any additional rights.

• No additional configuration is required. Configuration is done automatically by the LHO app.

• LHO uses the rights of the signed in user to enable monitoring of an entity.

• If the signed in user can modify the configuration of an entity in Databricks, then LHO also will be able to monitor that entity. Collecting telemetry data does not require any additional rights.

The telemetry collector agent is using the configured Secrets Scope (Azure Key Vault or Databricks Secrets) for securely accessing the cloud storage configuration. Databricks allows applications running inside Databricks environment to access secrets stored either in Databricks Secrets or Azure Key Vault.

LHO uses either Databricks Secrets or Azure Key Vault to provide credentials for the LHO Agent (collector) to use. The option is selected in the Settings page of the LHO app in the Databricks Workspace section, see “Created / Edit Secrets Scope” button.

A security administrator will need to create a Secrete Scope to store credentials for LHO Agent, by clicking “Create Secrets Scope”.

Once created, LHO app, in the Settings page > Access Rights, will be linked to the newly created Secret Scope. This means that LHO will be able to access and use credentials securely in a Databricks environment.

LHO App saves the following information in the configured Secret Scope:

• service principal

  • tenant id, client id, client secret

• storage account