Table of Contents

Step 1. Create Resource Group

Create the resource group where all the LHM dependent resources will reside.

1 Resource Group that contains the following:
- 1 Azure SQL Server:
  - Recommended Type: S3
- 1 Azure Key Vault:
  - Recommended Type: Standard
- 1 Azure Storage Account
- 1 Service Principal in Azure AD with “ID Token” enabled

resources description here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2532605979/Security+Requirements+for+VM+runtime#Phase-1)-Azure-Resources

Once all deployment steps are complete, the provided resource group will include the following resources:

Storage Account: used for storing all telemetry data from the Databricks workspaces and consumption/cost/usage detail data

KeyVault: used for storing the storage account access key (if enabled for the deployment), the Azure AD App Registration client secret (for Azure AD Single Sign On into the application and optionally for accessing the Azure Blob Storage account), and finally the SQL Server Login password.

SQL Server database: used for storing the output of the analyzer and consumption data processors, feeds all the data required by the reports and dashboards

Step 2. Create Storage Account

Inside that resource group create a storage account, with two file shares called

logs
keystore

Step 3. Create a KeyVault

more about secrets management here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#%F0%9F%97%84%EF%B8%8FSecrets-Storage

Step 4. Create a Azure SQL Server with Azure SQL Database

Create a Azure SQL Server and inside it create a Azure SQL Database

...

with SQL authentication

...

enabled.

Step 5. Create Database secret

The password you will save as a secret in the

...

Key Vault with the following secret name:mssql-password

more about secrets management here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#%F0%9F%97%84%EF%B8%8FSecrets-Storage

Step 6. Create a Azure App Registration

Create a Azure App Registration and fill in the Redirect URI with the following value:

Code Block
https://<dns label used below>.<location>.azurecontainer.io/login/oauth2/code/azure

see more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#How-do-I-configure-the-Azure-Active-Directory-Service-Principal%3F

Step 7. Edit App Roles

On the App Roles section in the Azure App Registration configuration panel, you can create the

...

LHM admin and

...

user roles which you can later use to control users access to the Lakehouse

...

Optimizer.

see more details here:https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#%F0%9F%AA%AA-LHM-Roles

Step 8. Create Client Secret

Create a client secret for your Application and save the secret in the KeyVault you’ve created with the following secret name msft-provider-auth-secret

see more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#How-do-I-configure-the-Azure-Active-Directory-Service-Principal%3F

Step 9. Create Managed Identity

Create a managed Identity that will be used by the Application to communicate to the keyvault and storage account.

LHM uses system-assigned managed identity
read more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#Managed-Identities

Step 10. Provide storage access to the SP

In order for the app to use the Service Principal to access the storage account you’ll need to:

Add the Contributor role to the Managed Identity you’ve created
1. Open the Storage Account
2. Go to Access Control (IAM)
3. Click on Add and follow the instructions to add the Contributor role to the Managed Identity
Add a permission mask on the container you’re going to be using in order for the Managed Identity to be able to write on it
1. From the Storage Account view, open the Containers section
2. Go to the container that will be used by LHM (the name of it is reflected within the STORAGE_AZURE_CONTAINER environment variable)
3. Go to Manage ACL
4. Click on Add principal and follow the steps to a mask for the Managed Identity
5. Make sure, once added, that you check all checkboxes for this new entry: Read, Write, Execute in order to give it access to use the container

Step 11. Edit Keyvault Access policies

Add the necessary access policy on the

...

Key Vault so the managed identity is able to list and read secrets from it.

In the Azure Portal, go to your KeyVault created in step 3.
In the left hand sidebar click on Access Policies
Image Added
Click on Create
From the Secrets section check List and Get as the permissions to be added
Search for the Managed Identity created on step 9 and select it. Click Next.
Skip the Application step as it’s optional
Click Create

Step 12. Configure SSL Communication

For SSL communication you need to create a keystore containing your server private key and certificate. The keystore will be named bplm.p12 and have no password.

Note: If this is not convenient you can

...

https://learn.microsoft.com/en-gb/azure/api-management/api-management-key-concepts

...

change it but you’ll need to change the environment variables below as well.

Command you can use to generate the keystore:

Code Block
openssl pkcs12 -export -out bplm.p12 -inkey ssl.key -in ssl.crt

Step 13. Create YAML definitions

Create YAML definition for the containers

Code Block

apiVersion: '2019-12-01'
location: <location>
name: <name of your container instance>
properties:
  containers:
  - name: lakehouse-monitor
    properties:
      environmentVariables:
      - name: APPLICATION_LOG_HTTPHEADER
        value: false
      - name: APPSERVICE_URL
        value: <FQDN>
      - name: AZURE_KEYVAULT_ENABLED
        value: true
      - name: AZURE_KEYVAULT_TENANTID
        value: <KV tenantID>
      - name: AZURE_KEYVAULT_URI
        value: <KV URL>
      - name: AZURE_MANAGED_IDENTITY_ID
        value: <managed identity objectID>
      - name: AZURE_MANAGED_IDENTITY_APP_ID
        value: <managed identity clientID>
      - name: LOG_LEVEL
        value: info
      - name: LOG_LEVEL_APP
        value: info
      - name: LOG_LEVEL_HTTP_HEADERS
        value: error
      - name: USE_SP_FOR_BACKGROUND_PROCESSORS
        value: true
      - name: USE_SP_FOR_STORAGE_ACCOUNT
        value: true
      - name: MICROSOFT_PROVIDER_AUTHENTICATION_SECRET
        value: ${msft-provider-auth-secret}
      - name: SERVICE_PRINCIPAL_CLIENT_SECRET
        value: ${msft-provider-auth-secret}
      - name: SERVICE_PRINCIPAL_CLIENTID
        value: <sp clientID>
      - name: SERVICE_PRINCIPAL_OBJECTID
        value: <sp objectID>
      - name: SERVICE_PRINCIPAL_TENANTID
        value: <sp tenantID>
      - name: SQL_DATABASE
        value: <sql database name>
      - name: SQL_SERVER_HOST
        value: <sql server host>
      - name: SQL_USER
        value: <sql username>
      - name: SQL_PASSWORD
        value: ${mssql-password}
      - name: STORAGE_AZURE_ACCOUNT
        value: <storage account name>
      - name: STORAGE_AZURE_CONTAINER
        value: lakehouse-monitor
      - name: CLOUD_PROVIDER
        value: azure
      - name: AUTHENTICATION_PROVIDER
        value: active-directory
      - name: SERVER_SERVLET_SESSION_PERSISTENT
        value: true
      - name: SERVER_SSL_ENABLED
        value:

...

true
      - name:

...

SERVER_SSL_

...

KEY_

...

STORE
        value:

...

/keystore/bplm.p12
#      - name:

...

SERVER_

...

SSL_

...

KEY_

...

STORE_

...

PASSWORD
#        value:

...

bplm
#      - name:

...

SERVER_SSL_

...

KEY_

...

ALIAS
#        value:

...

bplm
#      - name:

...

SERVER_SSL_

...

KEY_

...

PASSWORD
#        value:

...

bplm
      - name:

...

SERVER_SSL_KEY_STORE_TYPE
        value:

...

PKCS12

...

     - name:

...

METRIC_

...

PROCESSOR_

...

ENABLED
        value:

...

true
      - name:

...

METRIC_PROCESSOR_DOCTOR_INITIAL_DELAY
        value:

...

PT1M

...

- name: CONSUMPTION_USE_PREFILTER

...

   value: false

...

  - name: ADMIN_APP_ROLE

...

      value: <admin role

...

value>
      - name: EXECUTIVE_APP_ROLE

...

      value: <executive role value>

...

    - name: AUTHORIZATION_CACHE_TIMEOUT

...

value:

...

1800
      -

...

name:

...

SERVER_PORT

...

    value:

...

443
      image:

...

blueprint.

...

azurecr.

...

io/

...

bpcs/lakehouse-optimizer:2.3

...

ports:
      - port: 443
      -

...

port:

...

80
      resources:
        requests:
          cpu:

...

4
          memoryInGB:

...

8
      volumeMounts:
      - mountPath: /var/log
        name:

...

 logs
      - mountPath: /keystore

...

name:

...

keystore
  imageRegistryCredentials:
  - server: blueprint.azurecr.io
    username: <ACR Username>
    password: <ACR Password>
  osType: Linux
  restartPolicy: Always
  ipAddress:
    type: Public
    ports:
      - port: 443
      - port: 80
    dnsNameLabel: <dns label>
  volumes:
  - name:

...

logs

...

azureFile:

...

sharename:

...

<name of share for the logs>

...

storageAccountName:

...

<storage account name where the share is created>

...

storageAccountKey:

...

<storage

...

account

...

key>
  - name:

...

keystore
    azureFile:
      sharename: <name of share for the logs>
      storageAccountName: <storage account name where the share is created>
      storageAccountKey: <storage account key>
tags: {}
type: Microsoft.ContainerInstance/containerGroups

Step 14. Create Containers

Create the containers:

recommended minimum 8 cores

Code Block
az container create --resource-group <resource group name created in step 6> --assign-identity <managed identity to be used> --name <aci name> --file deploy.yaml

Versions Compared

Old Version 1

New Version Current

Key

Step 1. Create Resource Group

Step 2. Create Storage Account

Step 3. Create a KeyVault

Step 4. Create a Azure SQL Server with Azure SQL Database

Step 5. Create Database secret

Step 6. Create a Azure App Registration

Step 7. Edit App Roles

Step 8. Create Client Secret

Step 9. Create Managed Identity

Step 10. Provide storage access to the SP

Step 11. Edit Keyvault Access policies

Step 12. Configure SSL Communication

Step 13. Create YAML definitions

Step 14. Create Containers

Page Comparison

Versions Compared

Old Version 1

New Version Current

Key

Step 1. Create Resource Group

Step 2. Create Storage Account

Step 3. Create a KeyVault

Step 4. Create a Azure SQL Server with Azure SQL Database

Step 5. Create Database secret

Step 6. Create a Azure App Registration

Step 7. Edit App Roles

Step 8. Create Client Secret

Step 9. Create Managed Identity

Step 10. Provide storage access to the SP

Step 11. Edit Keyvault Access policies

Step 12. Configure SSL Communication

Step 13. Create YAML definitions

Step 14. Create Containers