2.25 Release Notes
The 2.25 release of Lakehouse Optimizer, includes major improvements in cost visibility, performance insights, and recommendation workflows. Here’s a summary of what’s new:
Key Updates
🔍 Top Recommendations Overhaul
Multiple UI/UX improvements, navigation fixes, and disclaimer additions for a smoother experience.
Performance boosted for large environments by optimizing data load and filtering.
Fixes to ensure accurate job and cost mapping between Top Recommendations and Workloads.
💰 Cost Analysis Enhancements
Cost breakdowns for workflow runs now persist in the database for consistent reporting.
Support for AWS CUR 1.0 introduced with proper permissions and warning logs if monthly data is missing.
Improved debug file structure and added the KB cost rate to SQL warehouse reports.
⚙️ Recommendation Engine Improvements
Better real-time recommendation generation and backfill behavior controls.
Exposed autopilot toggle in the UI and fixed estimation inconsistencies for job runs.
📈 Spend Insights & Forecasting Fixes
Updated budget and commit visuals in forecasting charts.
Improved configuration timeline UI and integrated it into the KPI selector.
Aligned and renamed labels for better clarity in spend charts.
🛠️ Infrastructure, Stability & UI Polish
Fixed issues related to job run cost analysis, missing durations, and incorrect status flags.
Enhanced cluster configuration views, including Photon capability and SQL warehouse runtime compatibility.
Improved UX details, including calendar closing logic, tooltip behavior, and “Copy Init Script” confirmations.
For a full list of enhancements and fixes, please refer to the detailed section below release documentation.
If you have any questions or feedback, we’d love to hear from you!
Detailed updates
🧠 Recommendation Engine Enhancements
Top Recommendations Revamp:
Navigation and UI improvements for better usability.
Fixed feedback-related issues and inconsistencies across environments.
Improved filtering for large-scale environments.
Step 1) View All recommendations
in all subscriptions and all workspaces
filtered by subscription and by workspace
Step 2) View Recommendations per entity (workflow, cluster)
Step 3) View Recommendations per single workflow run
Step 4) View Recommendations per cluster instance
grouped by cost saving or performance optimization
Each recommendation view provides more information related to the cluster instance configuration and the incidents related (root cause) that lead to the recommendation.
Each recommendation has one or more options to enhance performance or apply cost savings techniques.
Real-Time Improvements
Faster and more accurate recommendation generation.
Enabled backfill control and batch mode fixes.
Layer 2 and 4 updates including owner attribution and cost corrections.
Autopilot Visibility
Exposed Autopilot toggle in the UI for user control over automation behavior.
When Auto-Pilot is ON, new workloads will be detected automatically and be tracked by LHO.
Auto-Pilot Monitoring enhances the provisioning ability of Lakehouse Optimizer to automatically monitor ALL workloads in all published Databricks workspaces.
💰 Cost & Spend Insights Improvements
Workflow Run Cost Breakdown:
Costs are now stored in the database with detailed breakdowns, including duration fixes.
SQL Warehouse Cost Reporting:
KB cost rate now included.
Tag cost persistence improved to avoid duplication issues.
Compute uptime and idle ratio based on SQL Warehouse events (when available).
• Uptime is derived from RUNNING and STOPPED events.
• Active time is calculated from query history: the sum of durations of all queries executed on the warehouses.
• Idle time is computed as: idleTime = uptime - activeTime.
• when detecting which SQL Warehouse analyses need to be processed or reprocessed, we now check if new SQL Warehouse events are available.
• If no SQL Warehouse events are found (which is likely, since not all workspaces have the history store setting enabled), the analysis will include only the active time.
AWS Cost Support:
Added support for AWS CUR 1.0.
Introduced warnings when CUR data is missing for specific months.
Improved debug files with structured output and manifest handling.
“Untagged Databricks” Correction:
Serverless and real-time inference costs now excluded for better accuracy.
📊 Forecasting & Planning Enhancements
Spend & Budget Charts
UI fixes for commit and budget labeling.
Dialog alignment and month labels corrected.
Support commit values to edit/show for past years
KPI Selector
Added configuration timeline option.
Expanded control over visual data context.
⚡ Performance & Infrastructure Optimizations
Cluster & Job Run Analysis
Better runtime compatibility validation.
Photon capability visibility added.
Fixed infinite loops in job provisioning tasks.
Improved Data Flow
Queue prioritization for analysis tasks.
Support for controlling upsert vs insert behavior.
Batch sizing and grant retrieval optimizations.
🖥️ User Interface & Usability Updates
Navigation & Tooltips
Calendar now closes on outside click.
Tooltips and badges repositioned and realigned.
Shuffle size tooltip links fixed.
Enhanced visuals for configuration timeline
Trendline Tooltips: Each tooltip now displays duration data to aid performance review.
Visual Fixes
“Launched By” column refinements (rename, visibility toggle, default hidden).
One-Time Runs kebab menu repositioned.
Search bars and settings views redesigned for clarity.
search bar for workspace on the settings page moved closer to the list of workspaces
settings page expanded to support long workspace names
New Features
“Open in DBX” button added to Workflow Runs.
Attempt number included in single run URL paths.
Refresh-on-back for better browser navigation behavior.
Unity Catalog Migration updates
added batch size and artificial delay for the grants and extended table retrieval
display the latest run time and execution
Workloads / Job Run / Tasks export to CSV
🔍 Analysis, Insights & API Updates
Query History Enhancements
Bytes read/written and duration metrics now leveraged in serverless analysis.
API Additions
Job state API for last runs.
Cost rate exposure for incidents using estimated costs.
📊 Cost & Forecasting Improvements
Improved Incident Visibility: SKU cost incidents are now visible at the tenant level in the UI.
Enhanced accuracy of cloud cost tracking and reporting:
CostExplorer now verifies AWS role configuration, loading only from active accounts.
included monthly budget and monthly commit in spend insights kpi tooltip
🔧 Infrastructure & Stability Fixes
Azure VM & DBU Rates: Scripts for Azure VM and DBU prices were corrected to ensure accurate rate retrieval and updates.
Error Resilience: Improved task analysis during pipeline runs.
Made consumption loading more robust, ensuring failure on critical AWS usage errors and skipping finalization if needed.
This release delivers greater transparency, smarter recommendations, and a smoother user experience across all major components of Lakehouse Optimizer.