Azure Function for Lakehouse Optimizer Permissions setup script
- Denis Pitul
- Megan Novak
- Ciprian LUCACI
The following provides a comprehensive guide to understanding and utilizing the capabilities of the Azure Function for LHO Permissions setup script within your Azure environment.
This page contains:
script to grant permissions to Lakehouse Optimizer service principal
instructions to setup an automated mechanism to grant same permissions to LHO for newly added assets in a Databricks Workspace
The service principal used by LHO requires permissions to be granted to it in order to access both Databricks Workspaces and assets managed by workspaces. These access rights can be granted via the python_provisiong.py script, which requires an admin Databricks account to grant permissions.
If new assets (workflows, notebooks, clusters) are added in a workspace, then the LHO service principal needs permission to access those assets as well. This can be done automatically by adding the previous script as an Azure Function that runs recurrently on scheduled basis.
Prerequisites
Setup azcli
Install azcli core tool from Azure on your local machine
If core tools have been installed correctly, then func --version should work and return the current version.
Login with azcli
Run az login
How to create the Azure Function
Step 1. In the Azure Portal create a new resource group
This resource group is used to group all assets related to this Azure Function.
Step 2. Create a new function locally
Run the following in command line.
func init az_functions --python -m V1
cd az_functions
Step 3. Edit the requirements.txt file generated and append the following to it
certifi==2023.5.7
charset-normalizer==3.1.0
databricks-sdk==0.1.8
idna==3.4
requests==2.28.2
urllib3==1.26.16
Step 4. Create the storage account needed for the function
az storage account create --name lhmgrant --resource-group <YOUR_RESOURCE_GROUP> -sku Standard_LRS
Step 5. Create a new function based on the Timer trigger template
func new --name lhm-grant --template "Timer trigger"
Step 6. Download the python_provisioning.py and azure_main.py into the new function directory lhm-grant
Step 7. Edit azure_main.py and provide needed information for the grant code
Step 8. Change the cron schedule inside lhm-grant/function.json
func init will create a file function.json
Step 9. Change the lhm-grant/__init__.py file to call the grant function
import datetime
import logging
import azure.functions as func
from .azure_main import main as m
def main(mytimer: func.TimerRequest) -> None:
m()
Step 10. Create the Azure Portal FunctionApp
az functionapp create --resource-group <YOUR_RESOURCE_GROUP> --runtime python --runtime-version 3.7 --functions-version 4 --name lhm-grant --os-type linux --storage-account lhmgrant --consumption-plan-location <CONSUMPTION_PLAN_LOCATION>
Step 11. Publish the app
func azure functionapp publish lhm-grant