Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The Lakehouse Optimizer empowers users to monitor and improve their Lakehouse infrastructure by configuring incidents of interest that identify inefficiencies in cost, performance, and operational metrics. Incidents are displayed in the Incidents section of the app, providing actionable insights to optimize resource utilization and expenditures.

...

Incident Configuration Steps

  1. Under the Settings menu, select Settings / Incident Policies

    Incident Policies.png
    1. Incidents can be defined within Subscriptions, Workspaces, Workflows, All Purpose Compute, Delta Live Tables, SQL Warehouses, Pools, and Job Compute areas.

    2. Each incident created has its own incident policy

  2. Select the area of interest to create an incident under (ex. Workflow).

    Workflow.png


  3. Select category - Cost Control or Performance

    image-20250116-234155.png

  4. Select sub-category from dropdown menu (ex. Over Provisioning)

    Category and Sub Category.png


  5. Select sub-option within Sub-Category (ex. Cluster CPU Over Provisioning)

    Sub Category Details.png

...

  1. Every incident rule for a given Incident Policy automatically has email notifications turned on. However, by default no email group is tied to the incident rule, thus email notifications will not be sent until an email group(s) is selected for a given incident rule.

    Screenshot 2025-01-16 162211.png
  2. To assign an email group(s) to an incident rule, select the Email Group dropdown and select email group(s) (ex. Leadership Team, lho-support)

    Screenshot 2025-01-16 164535.png

...

Entity

Incident

Subscriptions

Monthly Cost above Threshold

Workspaces

Monthly Cost above Threshold

Workflows

Monthly Cost above Threshold

Workflows

  • Over-Provisioning

    • Cluster CPU

    • Driver Memory

    • Driver CPU

    • Driver Memory

Workflows

  • Under-Provisioning

    • Cluster CPU

    • Driver Memory

    • Driver CPU

    • Driver Memory

Workflows

  • Imbalanced-Provisioning

    • CPU

    • Memory

Workflows

Bad Skew

Workflows

Disk Spillage

Workflows

Run Failure

Workflows

Job with All Purpose Clusters

Delta Live Tables

Over Provisioning

  • Cluster CPU

  • Cluster Memory

Delta Live Tables

Under Provisioning

  • Cluster CPU

  • Cluster Memory

Delta Live Tables

Update Failure

Delta Live Tables

Monthly Cost above Threshold

All Purpose Clusters

Monthly Cost above Threshold

All Purpose Clusters

Auto Shutdown Timeout

  • Shutdown Timeout above Threshold

  • Shutdown Timeout Missing

All Purpose Clusters

Total Idle Time above Threshold

All Purpose Clusters

  • Over-Provisioning

    • Cluster CPU

    • Driver Memory

    • Driver CPU

    • Driver Memory

All Purpose Clusters

  • Under-Provisioning

    • Cluster CPU

    • Driver Memory

    • Driver CPU

    • Driver Memory

Pools

Auto Shutdown Timeout

  • Shutdown Timeout above Threshold

Email Group Configuration Steps

  1. To create a new email group, select Settings / Email Notifications

    Screenshot 2025-01-16 165951.pngImage Added
  2. Select +Add Group for Custom Groups

    image-20250117-010321.pngImage Added
  3. Specify group name (ex. Blueprint Test) and Save

    image-20250117-010405.pngImage Added
  4. Select +Add Email for the email group

    Screenshot 2025-01-16 170459.pngImage Added
  5. Specify an email account and Add

    Screenshot 2025-01-16 170943.pngImage Added
    1. Repeat process to include additional email accounts for a group

      Screenshot 2025-01-16 171234.pngImage Added
  6. See steps #8 and #9 under Incident Configuration Steps above on how to assign an email group to an incident policy.

Related articles