In order to monitor the Lakehouse Monitor we will be using existing Azure monitoring solutions (Insights and Alerts).
On the VM hosting your LHM you will be preparing a python environment and cron a python script that checks all the Subscriptions and Workspaces that are available to the Monitor in order to find any issues in retrieving information for them.
Script can be downloaded here <TODO create link>
| Code Block |
|---|
wget link to download |
Once the script is in place, it’s time to create the cronjob that runs it.
| Code Block |
|---|
crontab -e |
we recommend running the script every 30/60 minutes to be on top of the situation.
| Code Block |
|---|
0,30 * * * * python3 bplm-check.py |
The script and adjacent resources
Files present in the archive
bplm-check.py - main script
.env - environment file
requirements.txt - text file used to setup the python environment
Setting up the Application Insight
The script in the archive uses REST requests to query your LHM for information about: subscriptions, workspaces, etc.
Everything it does, including any issues it finds (no workspaces, no clusters, issues in connecting to them to list) it logs. The logs it sends to Azure. For this you’ll have to create a Application Insight in the Azure portal
in the portal, go to Monitor
in the left sidebar go to Insights → Applications
Create a Insight for your LHM deployment (in the same resource group as the other resources to ease locating it later)
...
Setting up the LHM check Azure function
Create a Function App by following the MS Azure docs here.
Make sure you create the Application Insights for the Function.
Create the system assigned managed identity for your Function App. You need this in order for the script to pull the client secret from the KeyVault.
Once the resources are created you can jump to creating the local function project part of the process (provided all the pre-requisites are met) you can create the function on your local machine by issuing the following:
| Code Block |
|---|
func init <PROJECT NAME> --python
cd <PROJECT NAME> |
Now you can pull the necessary files for the monitoring function:
| Code Block |
|---|
wget <URL TBD>
unzip -j bplm-check.zip |
The core files unarchived above is being defined by the following 4 files:
| Code Block |
|---|
bplm-check/bplmcheck.py
bplm-check/.env
bplm-check/__init__.py
requirements.txt |
the function is based on the TimeTrigger template and is configured to run every 30 mins. The cron expression can be found inside this file
| Code Block |
|---|
bplm-check/function.json |
At this point, you can publish the function to the Function App you’ve created above by issuing:
| Code Block |
|---|
func azure functionapp publish <FUNCTION APP NAME> -b remote |
Checking the Azure Function App at this point should show you the function you’ve just published.
...
Test it out by clicking on it, going to Code + Test and hit the Test/Run button. If you are not seeing the output in the console of this page, you can go to the Monitor page to see the logs. Successful runs look like this.
...
These log messages get publish to the Application Insights that’s tide to the Function App, which is where we’re setting up the alert based on the ERROR message count in these logs.
Setting up the alert
With the information successfully being sent to Azure, you can now set up an Alert.
...