2.1 Release Notes

Owned by Ciprian LUCACI
Last updated: Feb 02, 2023
11 min read

 

 

🎉 Highlights of 2.1 release

  • “Update Telemetry Now” - analyze on demand workflows that are in progress

  • Track SQL Warehouse costs per workspace

  • Health Alerts page

  • Improved scalability, performance and usability of Reports page when handling thousands of entities

Overview

  • User Story 4811: show percentages next to values - azure vs databricks cost in Overview distribution

  • User Story 4812: add cost distribution by percentage in the tooltips on overview page

  • cost breakdown percentages by cloud provider or Databricks

  • 4874: set default to "all workspaces" for right side widgets

  • 4504: update styling for overview page

  • 4706: workspace breakdown only aggregates on authorized workspaces when selecting 'All Workspaces'

  • 4569: show workspace id when hovering on the name of workspace on overview page in the costs breakdown widget

  • 4561: replace overview's cost telemetry alerts with Azure vs. Databricks Cost

  • 4525: track SQL Warehouse costs per workspace

  • 4682: update tooltip information on the cost breakdown widget to include information regarding SQL costs

  • 4566: compute total workspace cost by service with estimated costs

    • Cost data per workspace includes all entities present in a workspace, i.e. Workflows, All-purpose clusters, VM pools, Delta Live Tables, SQL Warehouses.

    • Expenses incurred by resources drawn from a pool are only considered proportionally to the amount of use in Job clusters or All-purpose clusters.

    • Total workspace cost is represented by cloud infrastructure costs (e.g. VMs, storage, network) and Databricks costs.

  • 4655: add highlight for current month cost on the subscription budget widget on the overview page

  • 4654: improve visuals on selectable rows on the cost breakdown widget in the overview page

  • 4653: make the widgets equal in width on the overview page

Documentation

  • 4829: updated titles and tooltips for all icons on the main menu to align with Databricks terms

  • 4638: add (i) icon next to Total Cost label to inform user about truncation

  • 4642: update tooltips on settings page

  • 4644: update info text on Reanalyze telemetry

Reports

  • 4797: improved timezone support when using date range filters

  • 4679: add azure vs databricks cost information on Reports

  • 4873: Add non dbu cost to job reports

  • 4623: use primary - apply and secondary - refresh for filters on redesigned Reports page

  • 4624: use primary - apply and secondary - refresh for filters on redesigned Reports // job runs details page

  • 4427: add cost & telemetry aggregations on Job overview with possibility to select the aggregated view

  • 4415: Hide shuffle size column on Reports jobs overview by default

  • 4501: Hide Sort by section on job runs

  • 4384: add results search on Job Runs

    • as a user you can search for a particular word or id in the results displayed on the screen

  • 4601: have TOTAL_COST as default sort by filter

  • 4612: Add in progress icon with tooltip on new reporting view

  • 4566: Compute total workspace cost by service with estimated costs. Fix unusedPoolsCost displaying N/A

  • 4664: Compute idleness also for clusters for which we don't collect any task metrics

  • 4688: Column tooltips on new reporting view for headers

On Demand Analysis

  • 3611: Trigger job run analysis (background service) that analyzes the run which is in progress

    • click a button to trigger telemetry analyzer on this job in order to get the telemetry data already available aggregated and prepared for reporting

  • 3901: Trigger only once at a time the "update telemetry" button per job run

  • 4745: Show update now button for pipelines that have SETTING_UP_TABLES or RUNNING status

  • 4622: hide the X on the sortBy filter until I hover over it

Security

  • 4448: Google security warning displayed. Fixed double redirect parameter issue

  • 4497: Support ServicePrincipal authentication to StorageAccount and Queue

  • 4573: Improve logic for checking if service principal is mandatory

  • 4652: ServicePrincipal Authorization filter in order to execute API calls using ServicePrincipal credentials

  • 4680: Secrets are not deleted using user credentials, but using Service Principal Managed Identity

  • 4661: Check service-principal/managed-identity should call ServicePrincipal scim API to verify if it's present in workspace, not list jobs.

  • support multiple Security Architecture Options

Authorization Rules

  • 4395: Clear Cached Access Rules button

    • Lakehouse Monitor caches for a limited amount of time the access rules for the logged-in user that are configured in Databricks in order to improve responsiveness. If you recently changed your access rules in Databricks, please clear the cached access rules. Lakehouse Monitor will refresh the access rules of the logged in user.

  • 4496: Add support for ActiveDirectory App Registration roles

  • 4659: Add an executive-role in LHM app

    • executive role can access all available workspaces

  • 4702: make Clear Cache function available to Users not only Admins

  • support Authorization Rules

  • support Public Subscriptions and Workspaces

  • 4749: add authorization and caching on reports/clusters for signed-in user

  • 4886: Optimize workspace retrieval in AlertsController. When user is executive, don't list workspaces

Performance Optimizations

  • 4386: Provide graceful shutdown for Consumption Loading

  • 4433: Refactor queue notification monitoring functionality in order to allow the monitoring of several jobs/pipelines in real time, without being limited by the size of the thread pool

  • 4561: Reduced prefilter max length due to the previous limit being too generous

  • 4685: Analyzer no longer checks if new metrics have been written

  • 4541: Pagination for Workflows details (job runs)

  • 4901: Reporter also starts a new analysis when updating the status of the existing analysis

  • 4860: async listing of the workspaces

  • 4749: add authorization and caching on reports/clusters for signed-in user

UX Optimizations

  • 4355: Authentication error not displayed in a friendly-mode

  • 4673: Increase fonts by at least 1px for names and column headers on Reports page

  • 4614: update navigation icons on the left bar

  • 4675: update color scheme and icons for the tabs in the Reports page

  • 4463: update badges for Workflows (jobs) for monitor status

  • 4258: Persist Date Range Filter as query params in the URL

    • &startDate=2023-01-12&endDate=2023-01-18

    • facilitates specific job run sharing

https://bplm-demo.westus.cloudapp.azure.com/job-runs/424988970744905?subscriptionId=a63c1e51-40ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7883487973999049.9.azuredatabricks.net&startDate=2023-01-12&endDate=2023-01-18

  • Bug 4721: workspace stats labels do not align properly and do not have proportional font size compared to the title tab

  • Bug 4722: improve color for workspace id on light theme on the cost breakdown on the overview page

  • Bug 4683: tooltip is outside of the screen if I navigate to the bottom right corner of the current month on the workspace stats widget in the overview page

  • Bug 4723: remove red color from the tooltip

  • 4850: Lowercase sort option name on reports page

  • 4846: cannot click on the links in the tooltip because the tooltip disappears too fast

❤️ Health Alerts

  • 4494: cost alerts api

  • 4574: Cost and Performance tabs

  • 4730: authorization layer for the health and cost alerts

  • 4597: when I click "open in reports" from health alerts, confiture the workspace in the reports detailed view

Operational Management – Telemetry & Consumption

  • 4762: Analyzer have to use the subscriptions and workspaces to which the SP has access.

  • 4253: Store job run analysis for all runs available in Databricks

  • 3769: Repair telemetry for jobs, notebooks and pipelines

  • 4462: Make 'Scheduled Analysis Intervals' section collapsable

  • 4568: Change the default number of Spark cores to be N - 2 for Telemetry Analysis engine

  • 4575: Remove Non Spark Workloads tab on Analysis History

  • 4590: Save signed-in user as "started by" info when triggering Run Now consumption jobs

  • 4590: Add startedBy field on consumption loading runs history

AWS support

  • use Dynamo DB instead of S3 as cloud storage solution for improved performance

  • use Azure AD Authentication provider in addition to Databricks identity provider with accounts API (username and password)

🎛️ Configuration

  • 4431: Add filter for "monitored" / "not monitored" when enabling or disabling the LHM agent

  • 4770: add support to enable collector using global init scripts from app

Deployment

  • PR 2470: Marketplace readiness for v.2.0.0

    • Changed how ssh key is generated and handled on VM to accommodate marketplace deployment

    • Re-enabled paramsoutput script to provide quick and easy argument list for Configure-lakehouseAD.ps1

    • Various changes to pass arm ttk "best practices"

  • 4550: Add API to verify if service principal is enabled for background processors

  • 4592: Retry mechanism for powershell deployment

  • 4670: Scripts to set up LHM on existing infrastructure
    adding install scripts

  • 4717: Added global init scripts templates to enable monitor

Miscellaneous

  • 4450 - Additional logic to resolve "unknown" workspaces

  • 4704 - Add endpoints to list clusters/jobs/pipelines using subscriptionId&workspaceId instead of workspaceResourceId

  • 4783: updated labels on Settings page for access rights and cache and sessions

 

🐞Fixed Bugs

  • Bug 4885: Duplicate notebook analysis due to a time race condition

  • Bug 4890: [UI] All Purpose monitor filter not working correctly

  • Bug 4878: Tooltips not visible on Cost breakdown when white theme is used

  • Bug 4881: On Reports page, the icons for column title dropdowns are not visible when white theme is used

  • Bug 4790: Misconsidered unused time sections

  • Bug 4877: Tab icons not visible on Reports page when white theme is used

  • Bug 4449: Empty cost data in Cost Tenant Subscription env

  • Bug 4458: Autoscaling segment too short for streaming jobs spanning multiple days

    • Removed the databricks based adjustments of start time, end time.

  • Bug 4455: Decrease size of background login image

  • Bug 4456: Unnecessary config request is made

  • Bug 4451: Overview - entity label spans 2 rows

  • Bug 4461: Miscalculate cluster termination status by cluster events

  • Bug 4465: Fix analysis task priorities

  • Bug 4388: Enable monitoring on all jobs warning - columns move when the page is changed

  • Bug 4389: Enable monitoring on all jobs warning - notification window changes size when the page is changed

  • Bug 4457: Sorting by index of VM cost distribution does not work

  • Bug 4467: Fix cpu & memory metrics for in progress job runs

  • Bug 4444: Incorrect cost reporting for pipeline updates that run over multiple days

  • Bug 4492: In progress run not displayed on job runs

  • Bug 4502: I click on the STOP button and nothing happens for consumption loading / run history

  • Bug 4511: Fix consumption loading steps status calculation

  • Bug 4485: Jobs with sql tasks display incorrect provisioning status

  • Bug 4532: Overview - actual data bars are not visible

  • Bug 4534: java.lang.ClassCastException thrown when repairing telemetry

  • Bug 4346: Task metrics of a notebook ran on a single node cluster don't get analyzed

  • Bug 4544: Fix cluster idleness cost start_time computation

    • Truncate cluster idleness start_time and end_time granularity to seconds

    • Consumption loading fail caused by cluster idleness cost primary-key error

  • Bug 4537: Duplicate job and pipeline analysis errors on demo

  • Bug 4542: Create/link secret scope should not enable “Already exists”

  • Bug 4543: Service Principal enabled should not say optional

  • Bug 4562: Fix for correct avg/max/total duration of the job runs

    • inconsistency in the reported duration for jobs

  • Bug 4570: Violation of PRIMARY KEY constraint on notebook analysis

  • Bug 4576: NPE exception when repairing workspace telemetry

  • Bug 457: Fix cloudshell deployment scenario

  • Bug 4586: Job reports page changes build - Sorting by name does not work

  • Bug 4591: Repair telemetry throws exceptions for SKIPPED runs

  • Bug 3747: Notebooks runs distribution is duplicated in case REDACTED_LOCAL_PART is present in the name of the notebook

  • Bug 4548: overview totals for December should be the same across the different classification, Provider, Resource or Service

  • Bug 4631: add missing colon after Jobs label

  • Bug 4632: Hide border in the tooltip when no selection

  • Bug 4621: Barcharts not proportional with cost

  • Bug 4636: [overview] When resizing the browser window, the X axis values on the graph extend outside the container

  • Bug 4650: Fix pagination for reports

  • Bug 4554: 403 Forbidden error when getting job details as regular user

  • Bug 4637: Missing empty validation for date range on runs view

  • Bug 4635: Update cache endpoint called on reports page even if the user didn't change the workspace

  • Bug 4542: Create/link secret scope should have “Already exists” checkbox as default to false

  • Bug 4236: Telemetry analysis details/Cluster idleness - date incorrectly displayed

  • Bug 4666: When going back from all purpose clusters detailed view to overview no get requests are made

  • Bug 4665: Pipeline updates reporting is not authorized

  • Bug 4727: Cluster idleness is not computed correctly

  • Bug 4731: NPE when managedIdentityAppId not configured and we run with service principal as background processors

  • Bug 4583: Secret scope button disabled for no reason

  • Bug 4738: Nested runs are not displayed as such

  • Bug 4734: Search is enabled when no workspace is selected

  • Bug 4584: Enable monitor on all jobs display empty list with jobs that cannot be monitorized

  • Bug 4633: consumption run history: switching page should scroll up and when reopening it it should be on page 1 again

  • Bug 4718: Skew Analysis Cannot Scroll

  • Bug 4743: Always open the run history on its first page

  • Bug 4403: Enable monitoring on all purpose-clusters seems successful when clusters are not actually monitored

  • Bug 4634: Filters bar scrolls off the screen while report view cuts off and has it's own scroll bar

  • Bug 4742: Change the error message displayed when the user is not authorized to access the runs of a particular job

  • Bug 4792: The view doesn't scroll up when switching pages in the Warning pop-up

  • Bug 4633: [UI] consumption run history: switching page should scroll up and when reopening it it should be on page 1 again

 

 

Last commit id: e5426885e