[WIP] Monitoring LHM

Setting up the LHM check Azure function

Create a Function App by following the MS Azure docs here.

Make sure you create the Application Insights for the Function.

Create the system assigned managed identity for your Function App. You need this in order for the script to pull the client secret from the KeyVault.

Once the resources are created you can jump to creating the local function project part of the process (provided all the pre-requisites are met) you can create the function on your local machine by issuing the following:

func init <PROJECT NAME> --python
cd <PROJECT NAME>

Now you can pull the necessary files for the monitoring function:

wget <URL TBD>
unzip -j bplm-check.zip

The core files unarchived above is being defined by the following 4 files:

bplm-check/bplmcheck.py
bplm-check/.env
bplm-check/__init__.py
requirements.txt

the function is based on the TimeTrigger template and is configured to run every 30 mins. The cron expression can be found inside this file

bplm-check/function.json

At this point, you can publish the function to the Function App you’ve created above by issuing:

func azure functionapp publish <FUNCTION APP NAME> -b remote

Checking the Azure Function App at this point should show you the function you’ve just published.

Test it out by clicking on it, going to Code + Test and hit the Test/Run button. If you are not seeing the output in the console of this page, you can go to the Monitor page to see the logs. Successful runs look like this.

These log messages get publish to the Application Insights that’s tide to the Function App, which is where we’re setting up the alert based on the ERROR message count in these logs.

Setting up the alert

With the information successfully being sent to Azure, you can now set up an Alert.

In the Application Insight go to the Logs section (left sidebar). All the logs go into the traces which is where you’ll want to query. Any error the script finds it will appear with a severityLevel=3 in this insight.

You can create use the following Kusto query to identify such issues

traces
| where severityLevel == 3

The results of this query can be counted afterwards to create an alert. The alert query becomes:

traces
| where severityLevel == 3
| count

At this point you need to click on the New Alert Rule button in the top bar to start creating your rule.

Set the alert logic to trigger when the value is bigger than 0 checking every 5 minutes.

Go to the next section “Actions”.

You will have to create an Action group here. Click on Create action group and fill in the necessary information. Go to the Notifications view and here you’ll want to create definitions for any notifications you want to be executed when an alert fires. E.g:

The actions view gives you the ability to execute any particular actions.

When the Action group is created, continue with the details page for your rule. Set the severity of the alert, by default it’s set to 3 - Informational. Give your rule a name and any additional details Azure might require. On the Advanced section you might consider enabling the option for “Automatically resolve alerts”. This way, when the alert condition is no longer met the alert will resolve on it’s own, this will weed out transient issues.

Once the rule is created and enabled, you’re all set up.