[WIP] Monitoring LHM

Setting up the LHM check Azure function

• Create a Function App by going to the Azure Portal, searching for “Function Apps” and clicking Create. Follow the rest of the wizard and make sure you create the Application Insights for the Function App as well (this will be needed in the alert creation stage later).

• Create the system assigned managed identity for your Function App by going to your Function app’s Identity section and setting the Status from Off to On. You need this in order for the script to pull the client secret from the KeyVault.

  

• Copy the Object ID from the identity page and go to your KeyVaults Access policies. Here you will create a new policy which will grant List and Get rights for Secrets for the Function Apps Identity (search for it by using it’s object ID.

• Next you will be creating the local function project. This part of the process takes place in your terminal.
  Make sure the pre-requisites are fulfilled. The following command creates the local project:

  func init <PROJECT NAME> --python # give it your function app name
  cd <PROJECT NAME>                 # the func init creates this folder

• After going into the function folder you will pull the necessary files for the monitoring function and unzip it in the current folder:

  wget <URL TBD>
  unzip -j bplm-check.zip

• The core files unarchived above is being defined by the following 4 files:

  bplm-check/bplmcheck.py
  bplm-check/.env
  bplm-check/__init__.py
  requirements.txt

  The other files in the archive are azure provided files which contain metadata pertaining to the monitor. They are:

  bplm-check/function.json
  bplm-check/host.json
  bplm-check/readme.md

  Note: the function is based on the TimeTrigger template and is configured to run every 30 mins. The cron expression can be found inside this file

  bplm-check/function.json

• Your current function folder should contain the following:

  .gitignore
  .vscode
  bplm-check
  getting_started.md
  host.json
  local.settings.json
  requirements.txt

Note: the bplm-check directory contains the files outlined above.

• At this point, you can publish the function to the Function App you’ve created above by issuing:

  func azure functionapp publish <FUNCTION APP NAME> -b remote

• Checking the Azure Function App at this point should show you the function you’ve just published.

• Test it out by clicking on it, going to Code + Test and hit the Test/Run button. If you are not seeing the output in the console of this page, you can go to the Monitor page to see the logs. Successful runs look like this.

Note: These log messages get publish to the Application Insights that’s tide to the Function App, which is where we’re setting up the alert based on the ERROR message count in these logs.

Setting up the alert

With the information successfully being sent to Azure, you can now set up an Alert.

In the Application Insight go to the Logs section (left sidebar). All the logs go into the traces which is where you’ll want to query. Any error the script finds it will appear with a severityLevel=3 in this insight.

You can create use the following Kusto query to identify such issues

traces
| where severityLevel == 3

The results of this query can be counted afterwards to create an alert. The alert query becomes:

traces
| where severityLevel == 3
| count

At this point you need to click on the New Alert Rule button in the top bar to start creating your rule.

Set the alert logic to trigger when the value is bigger than 0 checking every 5 minutes.

Go to the next section “Actions”.

You will have to create an Action group here. Click on Create action group and fill in the necessary information. Go to the Notifications view and here you’ll want to create definitions for any notifications you want to be executed when an alert fires. E.g:

The actions view gives you the ability to execute any particular actions.

When the Action group is created, continue with the details page for your rule. Set the severity of the alert, by default it’s set to 3 - Informational. Give your rule a name and any additional details Azure might require. On the Advanced section you might consider enabling the option for “Automatically resolve alerts”. This way, when the alert condition is no longer met the alert will resolve on it’s own, this will weed out transient issues.

Once the rule is created and enabled, you’re all set up.