Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

🎉 Highlights of 2.1 release

  • “Update Telemetry Now” - analyze on demand workflows that are in progress

  • Track SQL Warehouse costs per workspace

  • Health Alerts page

  • Improved scalability, performance and usability of Reports page when handling thousands of entities

🧭 Overview

  • 4504: update styling for overview page

  • 4569: show workspace id when hovering on the name of workspace on overview page in the costs breakdown widget

  • 4561: replace overview's cost telemetry alerts with Azure vs. Databricks Cost

  • 4525: track SQL Warehouse costs per workspace

  • 4682: update tooltip information on the cost breakdown widget to include information regarding SQL costs

  • 4566: compute total workspace cost by service with estimated costs

    • Cost data per workspace includes all entities present in a workspace, i.e. Workflows, All-purpose clusters, VM pools, Delta Live Tables, SQL Warehouses.

    • Expenses incurred by resources drawn from a pool are only considered proportionally to the amount of use in Job clusters or All-purpose clusters.

    • Total workspace cost is represented by cloud infrastructure costs (e.g. VMs, storage, network) and Databricks costs.

  • 4655: add highlight for current month cost on the subscription budget widget on the overview page

  • 4654: improve visuals on selectable rows on the cost breakdown widget in the overview page

  • 4653: make the widgets equal in width on the overview page

💾 Documentation

  • 4638: add (i) icon next to Total Cost label to inform user about truncation

  • 4642: update tooltips on settings page

  • 4644: update info text on Reanalyze telemetry

📋 Reports

  • 4427: add cost & telemetry aggregations on Job overview with possibility to select the aggregated view

  • 4415: Hide shuffle size column on Reports jobs overview by default

  • 4501: Hide Sort by section on job runs

  • 4384: add results search on Job Runs

    • as a user you can search for a particular word or id in the results displayed on the screen

  • 4601: have TOTAL_COST as default sort by filter

  • 4612: Add in progress icon with tooltip on new reporting view

  • 4566: Compute total workspace cost by service with estimated costs. Fix unusedPoolsCost displaying N/A

  • 4664: Compute idleness also for clusters for which we don't collect any task metrics

  • 4688: Column tooltips on new reporting view for headers

On Demand Analysis

  • 3611: Trigger job run analysis (background service) that analyzes the run which is in progress

    • click a button to trigger telemetry analyzer on this job in order to get the telemetry data already available aggregated and prepared for reporting

  • 3901: Trigger only once at a time the "update telemetry" button per job run

🔑 Security

  • 4448: Google security warning displayed. Fixed double redirect parameter issue

  • 4497: Support ServicePrincipal authentication to StorageAccount and Queue

  • 4573: Improve logic for checking if service principal is mandatory

  • 4652: ServicePrincipal Authorization filter in order to execute API calls using ServicePrincipal credentials

  • 4680: Secrets are not deleted using user credentials, but using Service Principal Managed Identity

  • 4661: Check service-principal/managed-identity should call ServicePrincipal scim API to verify if it's present in workspace, not list jobs.

  • support multiple Security Architecture Options

Authorization Rules

  • 4395: Clear Cached Access Rules button

    • Lakehouse Monitor caches for a limited amount of time the access rules for the logged-in user that are configured in Databricks in order to improve responsiveness. If you recently changed your access rules in Databricks, please clear the cached access rules. Lakehouse Monitor will refresh the access rules of the logged in user.

  • 4496: Add support for ActiveDirectory App Registration roles

  • 4659: Add an executive-role in LHM app

    • executive role can access all available workspaces

  • 4702: make Clear Cache function available to Users not only Admins

  • support Authorization Rules

  • support Public Subscriptions and Workspaces

🏎️ Performance Optimizations

  • 4386: Provide graceful shutdown for Consumption Loading

  • 4433: Refactor queue notification monitoring functionality in order to allow the monitoring of several jobs/pipelines in real time, without being limited by the size of the thread pool

  • 4561: Reduced prefilter max length due to the previous limit being too generous

  • 4685: Analyzer no longer checks if new metrics have been written

🎁 UX Optimizations

  • 4355: Authentication error not displayed in a friendly-mode

  • 4258: Persist Date Range Filter as query params in the URL

    • &startDate=2023-01-12&endDate=2023-01-18

    • facilitates specific job run sharing

https://bplm-demo.westus.cloudapp.azure.com/job-runs/424988970744905?subscriptionId=a63c1e51-40ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7883487973999049.9.azuredatabricks.net&startDate=2023-01-12&endDate=2023-01-18

❤️ Health Alerts

  • 4494: cost alerts api

  • 4574: Cost and Performance tabs

  • 4597: when I click "open in reports" from health alerts, confiture the workspace in the reports detailed view

⚙️ Operational Management – Telemetry & Consumption

  • 4253: Store job run analysis for all runs available in Databricks

  • 3769: Repair telemetry for jobs, notebooks and pipelines

  • 4462: Make 'Scheduled Analysis Intervals' section collapsable

  • 4568: Change the default number of Spark cores to be N - 2 for Telemetry Analysis engine

  • 4575: Remove Non Spark Workloads tab on Analysis History

  • 4590: Save signed-in user as "started by" info when triggering Run Now consumption jobs

  • 4590: Add startedBy field on consumption loading runs history

AWS support

  • use Dynamo DB instead of S3 as cloud storage solution for improved performance

  • use Azure AD Authentication provider in addition to Databricks identity provider with accounts API (username and password)

🎛️ Configuration

  • 4431: Add filter for "monitored" / "not monitored" when enabling or disabling the LHM agent

🖥️ Deployment

  • PR 2470: Marketplace readiness for v.2.0.0

    • Changed how ssh key is generated and handled on VM to accommodate marketplace deployment

    • Re-enabled paramsoutput script to provide quick and easy argument list for Configure-lakehouseAD.ps1

    • Various changes to pass arm ttk "best practices"

  • 4550: Add API to verify if service principal is enabled for background processors

  • 4592: Retry mechanism for powershell deployment

  • 4670: Scripts to set up LHM on existing infrastructure
    adding install scripts

  • 4717: Added global init scripts templates to enable monitor

🛠️ Miscellaneous

  • [4450] - Additional logic to resolve "unknown" workspaces

🐞Fixed Bugs

  • Bug 4449: Empty cost data in Cost Tenant Subscription env

  • Bug 4458: Autoscaling segment too short for streaming jobs spanning multiple days

    • Removed the databricks based adjustments of start time, end time.

  • Bug 4455: Decrease size of background login image

  • Bug 4456: Unnecessary config request is made

  • Bug 4451: Overview - entity label spans 2 rows

  • Bug 4461: Miscalculate cluster termination status by cluster events

  • Bug 4465: Fix analysis task priorities

  • Bug 4388: Enable monitoring on all jobs warning - columns move when the page is changed

  • Bug 4389: Enable monitoring on all jobs warning - notification window changes size when the page is changed

  • Bug 4457: Sorting by index of VM cost distribution does not work

  • Bug 4467: Fix cpu & memory metrics for in progress job runs

  • Bug 4444: Incorrect cost reporting for pipeline updates that run over multiple days

  • Bug 4492: In progress run not displayed on job runs

  • Bug 4502: I click on the STOP button and nothing happens for consumption loading / run history

  • Bug 4511: Fix consumption loading steps status calculation

  • Bug 4485: Jobs with sql tasks display incorrect provisioning status

  • Bug 4532: Overview - actual data bars are not visible

  • Bug 4534: java.lang.ClassCastException thrown when repairing telemetry

  • Bug 4346: Task metrics of a notebook ran on a single node cluster don't get analyzed

  • Bug 4544: Fix cluster idleness cost start_time computation

    • Truncate cluster idleness start_time and end_time granularity to seconds

    • Consumption loading fail caused by cluster idleness cost primary-key error

  • Bug 4537: Duplicate job and pipeline analysis errors on demo

  • Bug 4542: Create/link secret scope should not enable “Already exists”

  • Bug 4543: Service Principal enabled should not say optional

  • Bug 4562: Fix for correct avg/max/total duration of the job runs

    • inconsistency in the reported duration for jobs

  • Bug 4570: Violation of PRIMARY KEY constraint on notebook analysis

  • Bug 4576: NPE exception when repairing workspace telemetry

  • Bug 457: Fix cloudshell deployment scenario

  • Bug 4586: Job reports page changes build - Sorting by name does not work

  • Bug 4591: Repair telemetry throws exceptions for SKIPPED runs

  • Bug 3747: Notebooks runs distribution is duplicated in case REDACTED_LOCAL_PART is present in the name of the notebook

  • Bug 4548: overview totals for December should be the same across the different classification, Provider, Resource or Service

  • Bug 4631: add missing colon after Jobs label

  • Bug 4632: Hide border in the tooltip when no selection

  • Bug 4621: Barcharts not proportional with cost

  • Bug 4636: [overview] When resizing the browser window, the X axis values on the graph extend outside the container

  • Bug 4650: Fix pagination for reports

  • Bug 4554: 403 Forbidden error when getting job details as regular user

  • Bug 4637: Missing empty validation for date range on runs view

  • Bug 4635: Update cache endpoint called on reports page even if the user didn't change the workspace

  • Bug 4542: Create/link secret scope should have “Already exists” checkbox as default to false

  • Bug 4236: Telemetry analysis details/Cluster idleness - date incorrectly displayed

  • Bug 4666: When going back from all purpose clusters detailed view to overview no get requests are made

  • Bug 4665: Pipeline updates reporting is not authorized

Last commit id: 5262546d3

  • No labels