2.14.0 Release Notes

Owned by Ciprian LUCACI
Last updated: Apr 18, 2024
4 min read

 

 

✨ Release Highlights

  • Unity Catalog Assessment of migration readiness

    • evaluate how much of a Databricks workspaces have been migrated to Unity Catalog from Hive Mestastore

  • “Data Processed” KPIs exposed for Workflows

    • Data In, Data Out, Shuffle In, Shuffle Out

    • Data Processed = Data In + Data Out + Shuffle In + Shuffle Out

 

🔭 Unity Catalog Assessment

  • 9394 - Show the metastore and region on the Unity Catalog Migration page

  • 9469 - Assessment download buttons

image-20240418-190233.png

 

image-20240418-190255.png

 

🔦 Tenant Overview

  • 8898 - add month indicators to Optimization Health spark line

 

📁 Downloadable Reports

  • 9329 - short description for downloadable reports

 

Workspace Cost Overview

  • 8324 - add visual indicators for budget line, commit line and compute monthly value

 

Workloads // Workflows

  • 9330 - show "incidents count" in the workloads summary with aggregations

  • 9361 - Expose Spark task count in the data processed tooltip

  • 9340 - Ignore date filter when searching job runs by run id

  • 9331 - show "Unhealthy Runs" column in the workloads summary

    • runs with incidents identified

  • 9033 - Expose data metrics for workflows

    • Data In, Data Out, Shuffle In, Shuffle Out

    • Data Processed = Data In + Data Out + Shuffle In + Shuffle Out

 

❤️ Incidents // Performance

  • 9234 - No longer generate incidents for orphan nested notebooks

 

Performance Optimizations

  • 9314 - Added performances updates and timing tracking to notebook

    • Added performances for database and table evaluation. Based on initial testing, improved to 30 DBs a minute and 30 tables a minute

    • added new performance timing, integrated into database and table functions

    • additional error handling for DLT job objects

 

📊 Analysis Optimizations

  • 8369 - Update logic: Idleness and SQL Warehouse max worker count detection

    • optimize cluster idleness processing to load at most 2 days of events

    • optimize SQL Warehouse processing to find maximum cluster worker count using SQL query instead of loading all the cluster events

 

🪩 UX Optimizations

  • 9348 - added tooltip for tags management button

  • 9301 - enhance tooltip description and icons for Workloads overview

    • each tooltip contains also a description of the aggregation function used

      • cumulative, maximum, average, most recent

  • 9114 - created 'open in Databricks' option for pools in workloads

 

📈 Operational Management // Consumption (Cost) Analysis

  • 9390 - Handle empty 'dbus' and 'machineHours' fields as zeros for Billing Usages

 

📙 AWS Support

  • 9388 - deployment region is missing from AWS env

 

Miscellaneous

  • 8960 - One-time run job id icon not visible when white theme is used

 

🐞 Fixed Bugs

  • 9393 - Fix cluster info in recommendations for single node cluster

  • 9359 - Root cause and ticket ID missing from Drivers and Executors Incidents and Recommendations section