🎉 Highlights of 2.3.1 Release

• Estimated consumption value (cost) for current month based on previous consumption data

• Improved settings page to facilitate enhanced workspace configuration with clearer guidance and feedback

• Health Alerts panel with performance related alerts such as data skew, duration or shutdown policies

• Improved reporting of All Purpose Clusters with integrated autoscaling timeline and idleness analysis

• Settings page enhancements with enable/disable support for configuration options

• License scheme and editor with expiration policies

• Expose the list of LHM roles of the signed in user

Overview

• 4656 - Add predicted cost for current month

  • For the current month show also predicted cost besides the actual cost

  • Estimator takes into consideration all available cost data up to 24 months in the past

Open

actual and predicted cost for current month

• 5410 - Increase summary tiles pixel size for numbers to 18px on overview

Open

enlarged fonts for summary tiles

📂 Reporting // Workspaces

• 5492 - If a Databricks workspace was deleted, then also it is deleted from the LHM database

  • When an admin sign-in in the LHM app, and there are fewer workspaces retrieved from Databricks account than what LHM has in DB, then the database is updated to contain only workspaces found in the response from Databricks

  • This acts as sync mechanism to ensure LHM always reports up to date list of workspaces

Reporting // Workflows

• 5414 - Add end time to job run reports response

• 5498 - On cluster/jobs/pipelines rename, support order by name

  • ordering is done based on the latest names when telemetry was collected

🖥️ Reporting // All Purpose Clusters

• 5073 - improved clusters view with pagination and autoscaling timeline

• 4904 - provide creator of all-purpose-cluster when displaying cost of all purpose cluster

• 4671 - autoscaling timeline for clusters in the reports

• 5521 - support search by name for a cluster on Reports // All purpose clusters view

❤️ Health Alerts // Performance

• 5160 - Added performance alerts

  • event-based alerts for detection of

    • data skew

    • disk spillage

    • all purpose cluster shutdown timeout

    • pool shutdown timeout

Security

• 5601 - login with AWS admin credentials sends user to license screen

• 5206 - expose LHM roles of the user

  • more about roles here: Azure VM/Docker with Azure Service Management and Managed Identity | What roles are there in the LHM app?archived

📊 Analysis Optimizations

• 5221 - Use system load CPU instead of process load, when analyzing jobs/pipeline/notebook analysis

• 5424 - Telemetry Audit - Correlate the analysis detection with processing

  • the correlation between detection and processing is treated as one transaction, therefore we never purge an incomplete transaction even if it spans across more than one day

• 5558 - round-robin strategy for processing analyzer detected dbx assets (jobs, notebooks, dlts etc) due to large difference in volumes per different asset type

• 5615 - add a new section for Repair Cluster Idleness Analysis (in case configured threshold parameters for CPU thresholds are adjusted)

  • support updating the idleness detection if the algorithm has improved

🪩 UX Optimizations

• 5466 - Enable monitoring on all purpose cluster owned by other user fails with not formatted error message.

• 5131 - add tooltips in the titles of dialogs for cluster timeline and cost breakdown

🎛️ Configuration

• 5456 - pagination support and search on workflows provisioning page

• 5516 - Settings view improvements

  • 5101 - provide the ability to toggle on/off Deta Live Tables support

  • 5015 - provide the ability to disable GLOBAL INIT scripts and clean up all configurations related to LHM

  • 5016 - support upload / disable / update LHM Collector Agent

  • 5511 - on settings show badge per workspace for admin user that the workspace can be used by all users

  • 5510 - Settings page -> Search workspaces

• 5526: [ui] show workspace "configuration complete" for regular users that have set their Databricks token

• 5373 - support LHM license scheme

  • display warning that no license found

  • display warning when license close to expiration

Operational Management // Telemetry Analysis

• 5509 - Purge Audit logs older than a configurable number of days

  • Purge only successful audit runs older than a configurable number of days

📙 AWS Support

• 5528 - Optional skipping obtaining cloud costs from AWS CostExplorer

  • Use env config CONSUMPTION_AWS_SKIP_COST_EXPLORER=true to skip it

• 5281: UI: view/set cost-tags for storage and network

  • read more about resources tagging here: AWS Resource Requirements | Tags to activate in Cost Manager

Miscellaneous

• 5539 - Support for Unity enabled clusters and Databricks Access connectors (Managed Identities)

  • Agent now uses the Databricks runtime’s provided filesystem implementation for collecting telemetry into Blob Storage

• 4740: The eye for viewing resources should not be visible when the cost is NA

• 5565: add spark UI button for skew analysis

🐞 Fixed Bugs

• 5458 - Search box content should be reset when the tab is changed

• 5270 - Duplicate last months API requests when workspace changes

Documentation

• User Authentication with Active Directory Flow Diagram

  • User Authentication Flow Diagram with Active Directory in Azure

🎁 Coming Soon

SQL Warehouse reporting

Spillage Reporting with Memory Snapshot

Pools Activity View

… and other surprises 🎊