🎉 Highlights of this Release

• SQL Warehouse reporting

• Pool Cost breakdown

  • expose wasted resources as idling VMs

• Cluster Memory and CPU details

• Autoscaling timeline for clusters, jobs and notebooks running on all-purpose-clusters

• Detect Under/Over Provisioned resources for job, notebook, clusters

• Performance Evaluation Report:

  • Latency Score

  • Waste Score

  • Orchestration Score

Overview

• Pool resources cost breakdown

  • Idle VMs cost

    • expose wasted resources as idling VMs

  • Shared resources cost: network, disk, bandwidth

Open

Pool Cost breakdown on Azure

Open

Pool Cost breakdown on AWS

Reporting // Workflows

• expose Driver Memory and CPU and Cluster Nodes Memory and CPU data

  • enhances the investigation of under/over-provisioned workloads

  • provides the ability to detect skewed workloads with regards to memory and CPU on cluster nodes

🖥️ Reporting // All purpose clusters

• 5490 - All Purpose Cluster Configuration Timeline

• 5086 - improved feedback when the autoscale timeline is not found or not yet computed

• 5792 - expose autoscaling timeline for Notebooks and Jobs running on all-purpose-cluster

🖥️ Reporting // SQL Warehouse

• 5439 - SQL Warehouse analysis and reporting

• 5643 - Add cpu/memory information per driver and cluster information

❤️ Health Alerts // Cost

• 5827- support date range filter and sort by event type

❤️ Health Alerts // Performance

• 5895 - Added over/under provision alerts

  • job runs

  • notebook runs

  • clusters

🪩 UX Optimizations

• enhanced navigation by persisting filter selection in the URL in the Reports section

  • navigate back and forth by using the browser navigation bars

  • share link to your exact filter configuration

https://<YOUR_LHM_INSTANCE>/reporting?subscriptionId=a63c1e51-32ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7842187973999049.2.azuredatabricks.net&activeTab=1&clusterType=0&startDate=2022-05-12&endDate=2023-05-11&sortBy=TOTAL_COST&sortAscending=false&pageSize=25

Performance Optimizations

• 4779 - ability to REFRESH cache for user per subscription and workspace

• 5780 - clear cache for all users in the selected subscription and selected single workspace

📊 Analysis Optimizations

• 5721 - Fix the cluster idleness uptime calculation for clusters containing multiple app IDs within a single running segment

• 5749 - Make analysis detection stages independent

• 5621 - Enhance CPU-driven cluster idleness report by using task and executor metrics

• 5853 - support Repair Jobs Telemetry in Reanalyze Telemetry section

🎛️ Configuration

• 5350: limit user action on provisioning jobs or clusters and DLTs based on service principal and global init scripts configuration

Operational Management // Telemetry Analysis

• 5646 - Added correlationId to the diagnosisservice

• 5731 - startup logs with database stats (# subs, # workspaces, # jobs, # workspaces )

• 5208 - expose run frequency of the Health Alerts Detection Engine in the Operational Management section under Scheduled Analysis Intervals

Documentation

• 5633: add supporting information to tooltip when N/A shown for cost on overview pages

• 5639: updated app documentation for "billing admin"

• 5640: update "service principal" tooltip on the settings page