Step 1. Create Resource Group
Create the resource group where all the resources will reside.
more details here: Azure Resource Requirements
resources description here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2532605979/Security+Requirements+for+VM+runtime#Phase-1)-Azure-Resources
After the deployment of the Lakehouse Monitor is complete, the provided resource group will include the following resources:
VM includes the application services that provide the web interface and APIs for reporting and instrumentation dashboard, DBricks workspace configuration panel, background services for telemetry data analysis and recommendations, as well as consumption data scheduled runs
Storage Account: used for storing all telemetry data from the Databricks workspaces and consumption/cost/usage detail data
KeyVault: used for storing the storage account access key (if enabled for the deployment), the Azure AD App Registration client secret (for Azure AD Single Sign On into the application and optionally for accessing the Azure Blob Storage account), and finally the SQL Server Login password.
SQL Server database: used for storing the output of the analyzer and consumption data processors, feeds all the data required by the reports and dashboards
Step 2. Create Storage Account
Inside that resource group create a storage account
Step 3. Create a KeyVault
more about secrets management here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#%F0%9F%97%84%EF%B8%8FSecrets-Storage
Step 4. Create a Azure SQL Server with Azure SQL Database
Create a Azure SQL Server and inside it create a Azure SQL Database (use SQL authentication)
Step 5. Create Database secret
The password you will save as a secret in the keyvault with the following secret name: mssql-password
more about secrets management here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#%F0%9F%97%84%EF%B8%8FSecrets-Storage
Step 6. Create a Azure App Registration
Create a Azure App Registration and fill in the Redirect URI with the following value:
https://<dns label used below>.<location>.azurecontainer.io/login/oauth2/code/azure
see more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#How-do-I-configure-the-Azure-Active-Directory-Service-Principal%3F
Step 7. Edit App Roles
On the App Roles section you can create the bplm admin and executive roles which you can later use to control users access to the Lakehouse Monitor
todo: how ??? Costin Chiulan (Unlicensed) Denis Pitul
Step 8. Create Client Secret
Create a client secret for your Application and save the secret in the KeyVault you’ve created with the following secret name msft-provider-auth-secret
see more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#How-do-I-configure-the-Azure-Active-Directory-Service-Principal%3F
Step 9. Create Managed Identity
Create a managed Identity that will be used by the Application to communicate to the keyvault and storage account.
LHM uses system-assigned managed identity
read more details here: https://blueprinttechnologies.atlassian.net/wiki/spaces/BLMPD/pages/2571567105/Azure+VM+Docker+with+Azure+Service+Management#Managed-Identities
Step 10. Edit Access policies
Add the necessary access policy on the Key Vault so the managed identity is able to list and read secrets from it.
which policies ??? how?
Step 11. Configure SSL Communication
For SSL communication you can use one of these two options:
https://learn.microsoft.com/en-gb/azure/api-management/api-management-key-concepts
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-container-group-ssl (tested and used)
Step 12. Create YAML definitions
Create YAML definition for the containers
apiVersion: '2019-12-01' location: <location> name: <name of your container instance> properties: containers: - name: lakehouse-monitor properties: environmentVariables: - name: APPLICATION_LOG_HTTPHEADER value: false - name: APPSERVICE_URL value: <FQDN> - name: AZURE_KEYVAULT_ENABLED value: true - name: AZURE_KEYVAULT_TENANTID value: <KV tenantID> - name: AZURE_KEYVAULT_URI value: <KV URL> - name: AZURE_MANAGED_IDENTITY_ID value: <managed identity objectID> - name: AZURE_MANAGED_IDENTITY_APP_ID value: <managed identity clientID> - name: LOG_LEVEL value: info - name: LOG_LEVEL_APP value: info - name: LOG_LEVEL_HTTP_HEADERS value: error - name: USE_SP_FOR_BACKGROUND_PROCESSORS value: true - name: USE_SP_FOR_STORAGE_ACCOUNT value: true - name: MICROSOFT_PROVIDER_AUTHENTICATION_SECRET value: ${msft-provider-auth-secret} - name: SERVICE_PRINCIPAL_CLIENT_SECRET value: ${msft-provider-auth-secret} - name: SERVICE_PRINCIPAL_CLIENTID value: <sp clientID> - name: SERVICE_PRINCIPAL_OBJECTID value: <sp objectID> - name: SERVICE_PRINCIPAL_TENANTID value: <sp tenantID> - name: SQL_DATABASE value: <sql database name> - name: SQL_SERVER_HOST value: <sql server host> - name: SQL_USER value: <sql username> - name: SQL_PASSWORD value: ${mssql-password} - name: STORAGE_AZURE_ACCOUNT value: <storage account name> - name: STORAGE_AZURE_CONTAINER value: lakehouse-monitor - name: CLOUD_PROVIDER value: azure - name: AUTHENTICATION_PROVIDER value: active-directory - name: SERVER_SERVLET_SESSION_PERSISTENT value: true - name: SERVER_SSL_ENABLED value: false - name: METRIC_PROCESSING_ENABLED value: true - name: METRIC_PROCESSOR_DOCTOR_INITIAL_DELAY value: PT1M - name: CONSUMPTION_USE_PREFILTER value: false - name: ADMIN_APP_ROLE value: <admin role value> - name: EXECUTIVE_APP_ROLE value: <executive role value> - name: AUTHORIZATION_CACHE_TIMEOUT value: 1800 - name: SERVER_PORT value: 80 image: blueprint.azurecr.io/bpcs/lakehouse-optimizer:2.1 ports: - port: 80 resources: requests: cpu: 4 memoryInGB: 8 volumeMounts: - mountPath: /var/log name: logs - name: nginx-with-ssl properties: image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine ports: - port: 443 protocol: TCP resources: requests: cpu: 1.0 memoryInGB: 1.5 volumeMounts: - name: nginx-config mountPath: /etc/nginx imageRegistryCredentials: - server: blueprint.azurecr.io username: <ACR Username> password: <ACR Password> osType: Linux restartPolicy: Always ipAddress: type: Public ports: - port: 443 dnsNameLabel: <dns label> volumes: - name: nginx-config secret: ssl.crt: <BASE64 enc ssl.crt> ssl.key: <BASE64 enc ssl.key> nginx.conf: <BASE64 enc nginx.conf> - name: logs azureFile: sharename: <name of share for the logs> storageAccountName: <storage account name where the share is created> storageAccountKey: <storage account key> tags: {} type: Microsoft.ContainerInstance/containerGroups
Step 13. Create Containers
Create the containers
az container create --resource-group <resource group name created in step 6> --assign-identity <managed identity to be used> --name <aci name> --file deploy.yaml
Add Comment