Lakehouse Optimizer empowers users to monitor and improve their Lakehouse infrastructure by configuring incidents of interest that identify inefficiencies in cost, performance, and operational metrics. Incidents are displayed in the Incidents section of the app, providing actionable insights to optimize resource utilization and expenditures.
Steps for Incident Configuration
Under the Settings menu, select Settings / Incident Policies
Incidents can be defined for Subscriptions, Workspaces, Workflows, All Purpose Compute, Delta Live Tables, SQL Warehouses, Pools, and Job Compute.
Each incident created has its own incident policy
Select the area of interest to create an incident under (ex. Workflow).
Select category - Cost Control or Performance
Select sub-category from dropdown menu (ex. Over Provisioning)
The following incidents are configurable in LHO:
Entity | Incident |
|---|---|
Subscriptions | Monthly Cost above Threshold |
Workspaces | Monthly Cost above Threshold |
Workflows | Monthly Cost above Threshold |
Workflows |
|
Workflows |
|
Workflows |
|
Workflows | Bad Skew |
Workflows | Disk Spillage |
Workflows | Run Failure |
Workflows | Job with All Purpose Clusters |
Delta Live Tables | Over Provisioning
|
Delta Live Tables | Under Provisioning
|
Delta Live Tables | Update Failure |
Delta Live Tables | Monthly Cost above Threshold |
All Purpose Clusters | Monthly Cost above Threshold |
All Purpose Clusters | Auto Shutdown Timeout
|
All Purpose Clusters | Total Idle Time above Threshold |
All Purpose Clusters |
|
All Purpose Clusters |
|
Pools | Auto Shutdown Timeout
|
Add Comment