2.29 Release Notes
Sep 4, 2025
Lakehouse Optimizer 2.29.2
Audio Overview of Release
Cost Over Time
support weekly and daily cost breakdowns
support cost breakdown aggregated per week or per day
enhanced navigation from Forecasting page when investigating daily costs spikes
Cost Analysis
report AI Gateway cost in Cost Over Time breakdown Feature
Forecasting page
Weekly spend KPI chart
Forecasting: Cost & Data Usage chart: Avg. KPI value for the selected time period
each selected KPI has an averate for the selected period
Vendor Consolidation
improved documentation when data is not available, for N/A scenarios
Provisioning Workloads
map legacy_single_user_standard data mode to 'single'
Data Analysis
enhance orphaned task metrics resolving for job cluster analysis
improve performance on job runs trends page
add Data In, Data Out, Shuffle In, Shuffle Out in Edit table columns > Cluster instances page
Navigation
link RunJob Tasks to the actual Job definition
workflows that have tasks as RunJob include now a link from the task to the actual RunJob parent
Documentation
tooltip for "Auto Stop" column in SQL Warehouses table with link to Databricks article
Miscellaneous
updated design for job run statuses
list workspaces and subscriptions from DB instead of using cloud provider API
Aug 27, 2025
🚀 Lakehouse Optimizer 2.29.1
Dependent Vendor summary per job run and task
Updated design for exposing vendor informaion at task level and per job run.
The total data transferred and total wait time is the aggregation of unique Spark Job Groups metadata.
Data Transferred and Wait Time are extracted from Spark Job Group metadata. If multiple vendors are used simultaneously in the same Spark Job Group, then it is not possible to distinguish the direction and duration of data flow attributed to each unique vendor. In this scenario, the entire Spark Job Group’s values for data and time attributed to each vendor.
Support global date range across Workloads pages
once date is set on one page in workloads, the date will be passed also to all pages when navigating, unless there is another contraint, such as the exact date of a run
Enhanced documentation on Forecasting: Cost & Data Usage widget
add short description what is Data usage and Data Usage Cost
Dependent Vendor information for Notebooks
Dependent Vendor information for Pipelines
Job Runs Trendline KPIs documentation
each KPI includes now a description
Filter Migration Opportunities by Vendor
Forecasting : update CSV export feature to include all KPIs
Aug 20, 2025
🚀 Lakehouse Optimizer 2.29 – What's New
🔍 Cost and Forecasting Enhancements
support Enhanced Security and Compliance cost category to Databricks Platform
Cost Over Time > Databricks Platform > Enhanced Security and Compliance
Forecasting Weekly Fixes
Weekly cost KPIs and reports now follow ISO 8601 week numbering (Monday–Sunday), respecting user timezones. UI and CSV exports are now aligned.
Forecasting KPIs Improvements
Refined KPI groupings on the Forecasting page, improving clarity and structure for monthly views.
Cost Over Time CSV Fixes
CSV exports now include all relevant cost breakdowns. Improvements ensure consistency across breakdown levels and export formats.
KPI: Run Estimated Cost Visibility
The “Run Estimated” cost column is now always visible in Job Run views, helping users monitor cost projections directly.
KPI Calculations Fixes
Addressed several inconsistencies in forecasting KPIs and cost metrics (e.g., Cost per TB, Data Usage), including edge cases and null handling.
Improved Forecasting Tooltips and Layouts
Adjusted hover interactions and bar chart tooltips for clarity and responsiveness. Enhanced UI for forecasting side panels.
Breakdown for Other DBU/Cloud costs in Cost Over Time report
Display estimate badge for job cluster costs
A new badge visually flags when job cluster cost values are estimated, not actual.
Enable correlation for cost across pipelines with shared tasks
Enhanced cost attribution across pipelines with shared components for more accurate reporting.
Fix rounding issue in forecast chart
Spend Forecasts now display rounded values more precisely.
in Workloads: Jobs > Trend - Tasks add support for "Duration - Setup Only" and "Duration - Cleanup Only" KPIs
enables you to compare for each task how much it takes for the cluster instance to get setup or shutdown
🧠 Recommendations & Incidents
Persist job run recommendations
Recommendation history is now saved per job run, allowing users to track past optimization suggestions and results.
Fix search in historical recommendations
Historical recommendation queries now use the incident date instead of run start time, improving accuracy when searching.
Update DBX query for jobs with APCs incidents
Backend queries were refined for improved detection of incidents on All-Purpose Clusters (APCs).
Expose feature flag status for autopilot
Autopilot recommendation engine status is now visible to administrators for better monitoring and transparency.
Support rule override per workspace (Scope section)
Users can now override SKU Cost incident rules at the subscription or workspace level, giving greater control over incident thresholds and alerts.
Scope documentation for policy configuration panel
Recommendation Enhancements
The Disk Spillage recommendation now includes the spilled data size directly in its title for easier prioritization.
Autoscaling Timeline Link for Recommendations
Now available for Disk Spillage and Bad Skew insights, offering direct traceability to resource scaling events.
support Incident to compare daily cost of SQL Warehouse with AVG daily cost of Last Week, Last Month
📈 UI and UX Improvements
Unified Widget Proportions
Harmonized widget proportions on Forecasting and Vendor Consolidation pages for a more consistent and readable UI.
Session Expired Window
Improved UX for expired sessions with clearer messaging and visual indicators.
Cost Type Badges on Trend Views
Badges indicating Real/Estimated cost types were added or corrected across several trend pages for consistency.
Fix tab name when navigating from recommendations to workloads
Preserves the correct tab context during navigation for consistent user flow.
enhanced workloads navigation
a UI spacing fix improves visual alignment
add "Next" and "Back" words for page navigation with enabled and disabled states
Fix trendlines error for empty workloads
Prevented UI failures when loading trendlines for very short or idle workloads.
Improve dropdown display speed and styling
Faster dropdown responsiveness and improved layout consistency.
Fix double cursor pointer in search boxes
Resolved overlapping cursors in input fields for a cleaner experience.
🧮 Data Usage & Vendor Analytics
New Cost Categories
Added support for tracking Databricks Platform / Serverless GPU Compute and Enhanced Security & Compliance costs.
Data Usage Insights
Rolled out data usage statistics on cluster instance views and improved the “Cost of Data” KPI formula (now cost / data, using correct units).
AI & Model Training Cost Tracking
Foundation Model training costs and AI “other” expenses are now categorized and displayed correctly across reports.
Vendor Info at Task Level
Granular vendor metadata (including source database or cloud provider) now appears at the task level within workflows.
Vendors filtering on job runs
⚙️ Engine & Infrastructure Updates
Autoscaling & Serverless Adjustments
Enhanced logic for identifying serverless runs and computing proportional costs accurately across retries and multi-attempt jobs.
Job Config Timeline
Repaired inconsistencies in fetching job config snapshots for time-based analyses.
Query Metadata Expansion
Job query references now include notebook path, ADF flag, and start time, allowing for deeper insight into job context.
APC Cost Calculation Enhancements
Optimized how execution duration is factored into estimated APC (All-Purpose Compute) costs.
Fix issue when cluster_id is missing in runtime analysis
Resolved failures in job run analysis when certain cluster identifiers were unavailable.
Correct job run metadata when cluster name is missing
Ensures job runs without named clusters still retain accurate metadata.
Add DBX workspace ID to job runs table
Workspace IDs now appear in job run tables, supporting traceability across environments.
🧪 Stability and Bug Fixes
Add success criteria logs to help with debugging
Success criteria checks are now logged to simplify support investigations.
Efficiency Incident Validations
Fixed validation issues on incident rules related to excessive duration, idle time, and auto-shutdown parameters.
Visibility enhancements
Fixed incorrect default sort in Cost Controls.
Corrected layout spacing and alignment issues across Workflow Runs, Jobs Trend, and other views.
Adjusted tooltip spacing and icon visibility on multiple pages.
Technical Debt Reduction
Deprecated old reports API code.
Simplified cost pipeline computations.
Improved proportional cost calculations across shared clusters.