2.25 Release Notes

2.25 Release Notes

 

The 2.25 release of Lakehouse Optimizer, includes major improvements in cost visibility, performance insights, and recommendation workflows. Here’s a summary of what’s new:

 

Key Updates

🔍 Top Recommendations Overhaul

  • Multiple UI/UX improvements, navigation fixes, and disclaimer additions for a smoother experience.

  • Performance boosted for large environments by optimizing data load and filtering.

  • Fixes to ensure accurate job and cost mapping between Top Recommendations and Workloads.

 

💰 Cost Analysis Enhancements

  • Cost breakdowns for workflow runs now persist in the database for consistent reporting.

  • Support for AWS CUR 1.0 introduced with proper permissions and warning logs if monthly data is missing.

  • Improved debug file structure and added the KB cost rate to SQL warehouse reports.

 

⚙️ Recommendation Engine Improvements

  • Better real-time recommendation generation and backfill behavior controls.

  • Exposed autopilot toggle in the UI and fixed estimation inconsistencies for job runs.

 

📈 Spend Insights & Forecasting Fixes

  • Updated budget and commit visuals in forecasting charts.

  • Improved configuration timeline UI and integrated it into the KPI selector.

  • Aligned and renamed labels for better clarity in spend charts.

 

🛠️ Infrastructure, Stability & UI Polish

  • Fixed issues related to job run cost analysis, missing durations, and incorrect status flags.

  • Enhanced cluster configuration views, including Photon capability and SQL warehouse runtime compatibility.

  • Improved UX details, including calendar closing logic, tooltip behavior, and “Copy Init Script” confirmations.

 

For a full list of enhancements and fixes, please refer to the detailed section below release documentation.

If you have any questions or feedback, we’d love to hear from you!

 

Detailed updates

🧠 Recommendation Engine Enhancements

Top Recommendations Revamp:

  • Navigation and UI improvements for better usability.

  • Fixed feedback-related issues and inconsistencies across environments.

  • Improved filtering for large-scale environments.

 

Step 1) View All recommendations

  • in all subscriptions and all workspaces

  • filtered by subscription and by workspace

image-20250331-133352.png
all subscriptions and all workspaces
image-20250331-133424.png
filtered by subscription and by workspace

Step 2) View Recommendations per entity (workflow, cluster)

image-20250331-133642.png

Step 3) View Recommendations per single workflow run

image-20250331-133726.png

Step 4) View Recommendations per cluster instance

  • grouped by cost saving or performance optimization

image-20250331-133848.png
cost saving
image-20250331-133907.png
performance optimization

Each recommendation view provides more information related to the cluster instance configuration and the incidents related (root cause) that lead to the recommendation.

image-20250331-134432.png
cluster configuration
image-20250331-134627.png
root cause

Each recommendation has one or more options to enhance performance or apply cost savings techniques.

image-20250331-134510.png
options to solve

 

Real-Time Improvements

  • Faster and more accurate recommendation generation.

  • Enabled backfill control and batch mode fixes.

  • Layer 2 and 4 updates including owner attribution and cost corrections.

Autopilot Visibility

  • Exposed Autopilot toggle in the UI for user control over automation behavior.

    • When Auto-Pilot is ON, new workloads will be detected automatically and be tracked by LHO.

    • Auto-Pilot Monitoring enhances the provisioning ability of Lakehouse Optimizer to automatically monitor ALL workloads in all published Databricks workspaces.

image-20250331-133132.png
image-20250331-133204.png

 

 

💰 Cost & Spend Insights Improvements

Workflow Run Cost Breakdown:

  • Costs are now stored in the database with detailed breakdowns, including duration fixes.

SQL Warehouse Cost Reporting:

  • KB cost rate now included.

  • Tag cost persistence improved to avoid duplication issues.

image-20250331-141311.png

Compute uptime and idle ratio based on SQL Warehouse events (when available).
• Uptime is derived from RUNNING and STOPPED events.
• Active time is calculated from query history: the sum of durations of all queries executed on the warehouses.
• Idle time is computed as: idleTime = uptime - activeTime.
• when detecting which SQL Warehouse analyses need to be processed or reprocessed, we now check if new SQL Warehouse events are available.
• If no SQL Warehouse events are found (which is likely, since not all workspaces have the history store setting enabled), the analysis will include only the active time.

 

AWS Cost Support:

  • Added support for AWS CUR 1.0.

  • Introduced warnings when CUR data is missing for specific months.

  • Improved debug files with structured output and manifest handling.

“Untagged Databricks” Correction:

  • Serverless and real-time inference costs now excluded for better accuracy.

 

📊 Forecasting & Planning Enhancements

Spend & Budget Charts

  • UI fixes for commit and budget labeling.

  • Dialog alignment and month labels corrected.

  • Support commit values to edit/show for past years

image-20250331-132755.png
monthly budget and spend
image-20250331-132414.png
yearly commit editor
image-20250331-132443.png
budget editor

 

KPI Selector

  • Added configuration timeline option.

  • Expanded control over visual data context.

image-20250331-141550.png
image-20250331-141928.png

 

 

⚡ Performance & Infrastructure Optimizations

Cluster & Job Run Analysis

  • Better runtime compatibility validation.

  • Photon capability visibility added.

  • Fixed infinite loops in job provisioning tasks.

Improved Data Flow

  • Queue prioritization for analysis tasks.

  • Support for controlling upsert vs insert behavior.

  • Batch sizing and grant retrieval optimizations.

image-20250331-141807.png

 

 

🖥️ User Interface & Usability Updates

Navigation & Tooltips

  • Calendar now closes on outside click.

  • Tooltips and badges repositioned and realigned.

  • Shuffle size tooltip links fixed.

  • Enhanced visuals for configuration timeline

  • Trendline Tooltips: Each tooltip now displays duration data to aid performance review.

     

 

Visual Fixes

  • “Launched By” column refinements (rename, visibility toggle, default hidden).

  • One-Time Runs kebab menu repositioned.

  • Search bars and settings views redesigned for clarity.

    • search bar for workspace on the settings page moved closer to the list of workspaces

    • settings page expanded to support long workspace names

image-20250331-132937.png

 

New Features

  • “Open in DBX” button added to Workflow Runs.

  • Attempt number included in single run URL paths.

  • Refresh-on-back for better browser navigation behavior.

  • Unity Catalog Migration updates

    • added batch size and artificial delay for the grants and extended table retrieval

    • display the latest run time and execution

  • Workloads / Job Run / Tasks export to CSV

image-20250422-143726.png
image-20250422-143849.png

 

 

🔍 Analysis, Insights & API Updates

Query History Enhancements

  • Bytes read/written and duration metrics now leveraged in serverless analysis.

API Additions

  • Job state API for last runs.

  • Cost rate exposure for incidents using estimated costs.

 

📊 Cost & Forecasting Improvements

  • Improved Incident Visibility: SKU cost incidents are now visible at the tenant level in the UI.

  • Enhanced accuracy of cloud cost tracking and reporting:

    • CostExplorer now verifies AWS role configuration, loading only from active accounts.

  • included monthly budget and monthly commit in spend insights kpi tooltip

image-20250422-142444.png

 

 

🔧 Infrastructure & Stability Fixes

  • Azure VM & DBU Rates: Scripts for Azure VM and DBU prices were corrected to ensure accurate rate retrieval and updates.

  • Error Resilience: Improved task analysis during pipeline runs.

  • Made consumption loading more robust, ensuring failure on critical AWS usage errors and skipping finalization if needed.

     

 

This release delivers greater transparency, smarter recommendations, and a smoother user experience across all major components of Lakehouse Optimizer.