The Lakehouse Optimizer empowers users to monitor and improve their Lakehouse infrastructure by configuring incidents of interest that identify inefficiencies in cost, performance, and operational metrics. Incidents are displayed in the Incidents section of the app, providing actionable insights to optimize resource utilization and expenditures.
...
Incident Configuration Steps
Under the Settings menu, select Settings / Incident Policies
Incidents can be defined within Subscriptions, Workspaces, Workflows, All Purpose Compute, Delta Live Tables, SQL Warehouses, Pools, and Job Compute areas.
Each incident created has its own incident policy
Select the area of interest to create an incident under (ex. Workflow).
Select category - Cost Control or Performance
Select sub-category from dropdown menu (ex. Over Provisioning)
Select sub-option within Sub-Category (ex. Cluster CPU Over Provisioning)
...
Every incident rule for a given Incident Policy automatically has email notifications turned on. However, by default no email group is tied to the incident rule, thus email notifications will not be sent until an email group(s) is selected for a given incident rule.
To assign an email group(s) to an incident rule, select the Email Group dropdown and select email group(s) (ex. Leadership Team, lho-support)
...
Entity | Incident |
---|---|
Subscriptions | Monthly Cost above Threshold |
Workspaces | Monthly Cost above Threshold |
Workflows | Monthly Cost above Threshold |
Workflows |
|
Workflows |
|
Workflows |
|
Workflows | Bad Skew |
Workflows | Disk Spillage |
Workflows | Run Failure |
Workflows | Job with All Purpose Clusters |
Delta Live Tables | Over Provisioning
|
Delta Live Tables | Under Provisioning
|
Delta Live Tables | Update Failure |
Delta Live Tables | Monthly Cost above Threshold |
All Purpose Clusters | Monthly Cost above Threshold |
All Purpose Clusters | Auto Shutdown Timeout
|
All Purpose Clusters | Total Idle Time above Threshold |
All Purpose Clusters |
|
All Purpose Clusters |
|
Pools | Auto Shutdown Timeout
|
Email Group Configuration Steps
To create a new email group, select Settings / Email Notifications
Select +Add Group for Custom Groups
Specify group name (ex. Blueprint Test) and Save
Select +Add Email for the email group
Specify an email account and Add
Repeat process to include additional email accounts for a group
See steps #8 and #9 under Incident Configuration Steps above on how to assign an email group to an incident policy.