🚀 Lakehouse Optimizer 2.26 – Enhanced Cost Accuracy, UI Performance & System Table Support

We’re excited to introduce Lakehouse Optimizer v2.26, a feature-packed release focused on cost precision, performance tuning, and enhanced visibility across your workloads and recommendations.

We recommend reviewing these updates in your environment to take full advantage of the improvements in cost transparency, performance insights, and operational stability.

If you have any questions, feel free to reach out to your support contact or consult the updated documentation.

Below is a summary of the key improvements:

💰 Cost Tracking & Serverless Billing Enhancements

Accurate Serverless Rates: Rates for serverless compute are have been updated.
Missing Costs Fixed: Resolved issues with untracked serverless costs and DLT cluster estimates.
Improved Cost Grouping: Unrecognized DLT costs with missing pipeline_id are now assigned to an “Other” category, preserving reporting integrity.
Added a “Cost of Data” metric and renamed columns to “Data Processed” in SQL Warehouse views.
Added serverless-specific KPIs such as Data Processed, Hourly Cost Rate, and Cost of Data.

SQL Warehouses / data processed

SQL Warehouses / cost of data

Serverless-specific KPIs such as Data Processed, Hourly Cost Rate, and Cost of Data

Improved pipeline cost breakdown, including:
- Cluster instance run cost attribution.
- Run cost computation for tasks with multiple attempts.
- Support for DLT and serverless pipelines.
Exposed SQL Warehouse uptime, active/idle time, and efficiency KPIs.

SQL Warehouse uptime, active/idle time

Added support for estimated and real costs to be displayed side-by-side in trendlines.

“Serverless SQL Databricks Cost” is now renamed to “Serverless SQL Warehouse” for clarity.
Applied rounding to VM rate display and fixed discrepancy in cost breakdown charts under $1.
Added cost metadata API for DBX and Cloud charges including DBCU discount plans and sources.

Cloud Cost Data

Databricks Cost Data

Export to CSV support

Spend Insights / Forecasting / Monthly / Export Function to CSV

export to CSV

export confirmation

monthly KPIs - sample export

Workloads // All-Purpose Compute > Clusters > Export to CSV

All-Purpose Compute Export to CSV

🔔 Cost Control Notifications

The daily SKU Cost incident policy detects sudden cost spikes for Databricks resources (e.g., APCs, jobs, “Other Resources”) by comparing a day’s cost to both the prior full week and a fixed 30-day sliding window.

Incidents are triggered when the cost deviates beyond a configurable threshold—by percentage or absolute value—and evaluated separately for workspace-level and tenant-level aggregations.

Workspace-level incidents support scoped rules: users can override global thresholds by defining custom rules at the subscription or workspace level, with narrower scopes taking precedence.

For example, a workspace-specific rule overrides a subscription rule, which in turn overrides the global rule. In contrast, tenant-level incidents only support global rules, as costs are aggregated across the entire tenant. This flexible scoping model ensures precise and context-aware alerting tailored to different organizational needs.

support daily SKU Cost incidents (for workspace and tenant aggregation level) for "Other Resources". This includes assets that don't fall in any predefined resource category.
Exposed detailed problem descriptors in incident APIs.

set incidents policies for multiple types of workloads, features or resources types

For example, you can now monitor All-Purpose Compute either by Individual cluster or track the entire cost associated with All-Purpose Compute for the entire tenant or by workspace.

All-Purpose Compute / Cost Control policies

Create new policy to track All-Purpose Compute costs

All-Purpose Compute cost per workspace / cost increase incidents

🖥️ User Experience Improvements

Performance Boost: Top Recommendations view now loads significantly faster in large environments.
UI Polishing: Fixes across tooltips, filters, date formats, and navigation from incidents to workloads and recommendations.
- bug: Incorrect pointer cursor on non-links - Incidents
- bug: selector search bar automatically selects first found item
- Applied visual updates like elevation for filter panels, badge alignment, and color consistency.
Timezone Accuracy: Dates such as job start times and incident event times now use consistent formatting and user timezone awareness.
Optimized Top Recommendations view for faster load time and better responsiveness.
Trendlines now display tooltip durations and support outlier/failed job filtering.

🧠 Incident & Recommendations Logic

Introduced incident scope overrides for SKU Cost incidents.

Users can now define SKU Cost incident rules at:

Subscription level (applies to all workspaces in that subscription)
Workspace level (overrides both subscription and global rules)
Rule priority: Workspace > Subscription > Global
Disabled rules fall back to the next broader scope.
Scope is only available for workspace-level aggregated incidents.
Tenant-level and Individual Asset incidents remain global-only.
- In the UI:
  - Scope selection is shown only where supported.
  - Subscription must be selected before workspace.
  - Rule scope cannot be changed after creation.

policy applicable for all workspaces in all subscriptions

policy applicable only for the selected workspace

Updated incident names:
- “Excessive Duration” ➝ “Cluster Instance Duration Exceeded” for APC.
- DLT incidents now prefixed with “Pipeline” to distinguish from APC.
Added badges to inform users when incidents are based on estimated costs.

Enhanced navigation to recommendations from on Workflow Runs and Cluster Instances.

navigate to recommendations for a selected workflow

navigate to recommenations per job run

navigate to recommenations per cluster instance

🧠 Analysis & Incident Intelligence

Smarter Incident Detection: Improved rules for Job-with-APC and SKU cost incidents, and removed unnecessary notifications.
Notebook Filtering: Reports now exclude non-serverless notebooks, and avoid double-counting metrics for SQL Warehouses.
Introduced a new Job Duration Exceeded incident type.
More Precise UI Calculations: Rounded VM rate displays to two decimals and improved filtering logic.
Percentage Change Fix: Now accurately reflects meaningful differences even between small dollar amounts (e.g., <$1 vs $2).
Forecasting & UI Fixes: Removed unnecessary totals and fixed year selection behavior in budget and commit configuration.

🔬 System Security, Reliability & Improvements

Support for System Schema Tables: Enabled provisioning, analysis, and integration test coverage for system schema tables.
Improved Scheduler Logic: Fixed issues with scheduler disabling and SQL warehouse event types (e.g., STARTING vs RUNNING) for uptime tracking.
Stability Updates: Telemetry edge cases (like missing correlationId) handled.
API & Backend: Updates to support pipeline-level cost breakdowns and improved job run attempt handling in cost reporting.
Enabled forecaster auto-deployment on new environments.
Updated Docker volume isolation to prevent disk saturation.
Add support for temporary security credentials for Agent to access DynamoDb/Sqs
Panel in the System Updates section with the LHO status

LHO status panel