2.24 Release Notes
- Ciprian LUCACI
The latest Lakehouse Optimizer 2.24 update brings enhancements in cost tracking, performance optimizations, incident reporting, and usability improvements.
This release brings improvements in performance tracking, cost visibility, and usability, ensuring a more streamlined and efficient experience for Lakehouse Optimizer users.
Key updates
📊 Cost & Spend Management Enhancements
Cluster cost transparency: Added a disclaimer for All-Purpose Compute cluster costs, clarifying that storage and networking are not included.
Better forecasting insights: Forecasting report KPIs now appear as tooltips on the main page for quicker access.
Workspace Cost Fixes: Resolved an issue where search by name did not work when Caps Lock was enabled.
SQL Warehouse Analysis Enhancements: Improved execution time deduplication and analysis processing for SQL workloads.
include deleted SQL warehouses if they have associated costs
⚡ Performance & Workflow Optimization
Incident Generation in Real-Time: Job over/under/imbalanced provisioning incidents now generate immediately for better resource allocation.
SQL Warehouse Execution Enhancements: Added indexes to improve job/pipeline configuration timelines and reduce read load.
Workflows Cost Tracking: Adjustments made to better track DLT, RunJob, and Serverless costs* in workflow run cost analysis.
Trendline KPIs Added: New Processed GB/TB per Hour Rate KPIs for more insightful trend analysis.
Improved tracking and display of SQL Warehouse tag costs, optimizing cost reporting.
🚨 Incidents & Recommendations
Real-Time Recommendations: Implemented real-time generation of recommendations for bad skew and disk spillage to enhance performance.
Trendline Usability Enhancements: A new expandable panel provides guidelines for trendline analysis, helping users select optimal workflow runs for cost and performance improvements.
Job Failure Incident Tracking: Added a detailed failed task list for job run failure incidents to improve troubleshooting.
Updated job run failure incident generation to occur in real-time instead of per-task.
🖥️ UI & Usability Improvements
Improved Workflow Exporting: Users can now export Workloads Overview and Job Runs pages to CSV for analysis.
Consistent UI Elements: The three-dot menu is now always visible, and clickable IDs are underlined for a more intuitive experience.
UI Enhancements: Display an “estimated” badge when estimated costs are enabled in Spend Insights and Cost by Month views.
📄 Documentation & Miscellaneous Fixes
New Documentation for Purge/Telemetry Data: Guidelines added for removing telemetry data from Azure Storage to optimize cloud costs.
Bug Fixes:
Fixed timezone issues affecting cost control incidents.
Addressed an issue where Init script failure notifications were not sent.
Improved SQL Warehouse cost correlation for better accuracy.
Resolved UI glitches with analysis history and updates pages.
Detailed updates
📁 Downloadable Reports
Export to CSV the current page for Jobs Overview and Job Runs
the CSV export feature prepares the current page as a CSV file ready to be downloaded
set more items per page if you want more entries to be added in the exported file
🏦 Spend Insights / Forecasting
show forecasting report KPIs as tooltip information on main page
Display an “estimated” badge when estimated costs are enabled in Spend Insights and Cost by Month views.
Improved cost breakdowns: Fixed discrepancies in DBX vs. Cloud costs for the ‘Other’ cost category.
✅ Incidents & Recommendations
Enable real time generation recommendations for bad skew and disk spillage
add trendline guideline as expandable panel
Select a run that is representative of this workflow as a reference point for optimization recommendations. Any recommendations applied for this reference run will affect future workflow runs. Before selecting the run, analyze the trend lines to learn how the runs of this workflow compare to each other in terms of data processed (data in/out or processed during the run), CPU/memory utilization, cost etc.
If runs are similar, than any recent run this period can be used as reference point for optimization. If not, we recommend creating multiple workflow definitions to allow for clustering these runs in their own Workflow with the same definition but different schedules. This separation of concerns would allow you to configure them differently to meet your business needs.
For example, the same workflow could be used to process different data size depending on time of the day (AM vs PM), or different data source or different workflow parameters that would ultimately lead to different workflow run behaviors and different recommendations from LHO.
Workloads // Workflows
add Processed GB/TB per Hour Rate as new KPIs for Trendlines
add disclaimer for cost of cluster instances
Cost of All-Purpose Compute cluster instances is an estimated cost that does not include the the cost of disk storage and networking.
🖥️ Workloads // Cluster Instances
enhanced panel for cluster instance configuration
12469 - add disclaimer for cost of cluster instances
Performance Optimizations
use job cluster analysis instead of job run analysis for optimizing histogram requests
update success criteria query to account for the data coming from workflows
generate job over/under/imbalanced provisioning incidents in real time
add indexes to improve job/pipeline configuration timelines
database performance enhancements by reducing read load
🪩 UX Optimizations
make 3 dots always visible and underline clickable IDs
Miscellaneous
warn user that whitespaces are not supported for free text search
improve cost per data calculation in reports
fix scenario when Init script failure notification email is not sent
fixed timezone issue for cost control incidents
fixed Search by name not working when Caps Lock is on for Workspace Cost
Documentation
add documentation for Purge / Telemetry Data
Remove telemetry data saved in Azure Tables in your Azure Storage account .
Removing old data that is no longer relevant optimizes cloud storage cost.
