2.4 Release Notes

Owned by Ciprian LUCACI
Last updated: May 12, 2023 by Claudiu Barbura
4 min read

 

🎉 Highlights of this Release

  • SQL Warehouse reporting

  • Pool Cost breakdown

    • expose wasted resources as idling VMs

  • Cluster Memory and CPU details

  • Autoscaling timeline for clusters, jobs and notebooks running on all-purpose-clusters

  • Detect Under/Over Provisioned resources for job, notebook, clusters

  • Performance Evaluation Report:

    • Latency Score

    • Waste Score

    • Orchestration Score

 

Overview

  • Pool resources cost breakdown

    • Idle VMs cost

      • expose wasted resources as idling VMs

    • Shared resources cost: network, disk, bandwidth

 

Pool Cost breakdown on Azure
Pool Cost breakdown on AWS

 

Reporting // Workflows

  • expose Driver Memory and CPU and Cluster Nodes Memory and CPU data

    • enhances the investigation of under/over-provisioned workloads

    • provides the ability to detect skewed workloads with regards to memory and CPU on cluster nodes

 

🖥️ Reporting // All purpose clusters

  • 5490 - All Purpose Cluster Configuration Timeline

  • 5086 - improved feedback when the autoscale timeline is not found or not yet computed

  • 5792 - expose autoscaling timeline for Notebooks and Jobs running on all-purpose-cluster

 

🖥️ Reporting // SQL Warehouse

  • 5439 - SQL Warehouse analysis and reporting

  • 5643 - Add cpu/memory information per driver and cluster information

 

❤️ Health Alerts // Cost

  • 5827- support date range filter and sort by event type

 

❤️ Health Alerts // Performance

  • 5895 - Added over/under provision alerts

    • job runs

    • notebook runs

    • clusters

 

🪩 UX Optimizations

  • enhanced navigation by persisting filter selection in the URL in the Reports section

    • navigate back and forth by using the browser navigation bars

    • share link to your exact filter configuration

https://<YOUR_LHM_INSTANCE>/reporting?subscriptionId=a63c1e51-32ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7842187973999049.2.azuredatabricks.net&activeTab=1&clusterType=0&startDate=2022-05-12&endDate=2023-05-11&sortBy=TOTAL_COST&sortAscending=false&pageSize=25

 

Performance Optimizations

  • 4779 - ability to REFRESH cache for user per subscription and workspace

  • 5780 - clear cache for all users in the selected subscription and selected single workspace

 

📊 Analysis Optimizations

  • 5721 - Fix the cluster idleness uptime calculation for clusters containing multiple app IDs within a single running segment

  • 5749 - Make analysis detection stages independent

  • 5621 - Enhance CPU-driven cluster idleness report by using task and executor metrics

  • 5853 - support Repair Jobs Telemetry in Reanalyze Telemetry section

 

🎛️ Configuration

  • 5350: limit user action on provisioning jobs or clusters and DLTs based on service principal and global init scripts configuration

Operational Management // Telemetry Analysis

  • 5646 - Added correlationId to the diagnosisservice

  • 5731 - startup logs with database stats (# subs, # workspaces, # jobs, # workspaces )

  • 5208 - expose run frequency of the Health Alerts Detection Engine in the Operational Management section under Scheduled Analysis Intervals

 

Documentation

  • 5633: add supporting information to tooltip when N/A shown for cost on overview pages

  • 5639: updated app documentation for "billing admin"

  • 5640: update "service principal" tooltip on the settings page