Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

🎉 Highlights of 2.1 release

  • “Update Telemetry Now” - analyze on demand workflows that are in progress

  • Track SQL Warehouse costs per workspace

  • Health Alerts page

  • Improved scalability, performance and usability of Reports page when handling thousands of entities

🧭 Overview

  • 4504: update styling for overview page

  • 4569: show workspace id when hovering on the name of workspace on overview page in the costs breakdown widget

  • 4561: replace overview's cost telemetry alerts with Azure vs. Databricks Cost

  • 4525: track SQL Warehouse costs per workspace

  • 4682: update tooltip information on the cost breakdown widget to include information regarding SQL costs

  • 4566: compute total workspace cost by service with estimated costs

  • 4655: add highlight for current month cost on the subscription budget widget on the overview page

  • 4654: improve visuals on selectable rows on the cost breakdown widget in the overview page

  • 4653: make the widgets equal in width on the overview page

💾 Documentation

  • 4638: add (i) icon next to Total Cost label to inform user about truncation

  • 4642: update tooltips on settings page

  • 4644: update info text on Reanalyze telemetry

📋 Reports

  • 4427: add cost & telemetry aggregations on Job overview with possibility to select the aggregated view

  • 4415: Hide shuffle size column on Reports jobs overview by default

  • 4501: Hide Sort by section on job runs

  • 4384: add results search on Job Runs

  • 4601: have TOTAL_COST as default sort by filter

  • 4612: Add in progress icon with tooltip on new reporting view

  • 4566: Compute total workspace cost by service with estimated costs. Fix unusedPoolsCost displaying N/A

  • 4664: Compute idleness also for clusters for which we don't collect any task metrics

  • 4688: Column tooltips on new reporting view for headers

On Demand Analysis

  • 3611: Trigger job run analysis (background service) that analyzes the run which is in progress

  • 3901: Trigger only once at a time the "update telemetry" button per job run

🔑 Security

  • 4448: Google security warning displayed. Fixed double redirect parameter issue

  • 4497: Support ServicePrincipal authentication to StorageAccount and Queue

  • 4573: Improve logic for checking if service principal is mandatory

  • 4652: ServicePrincipal Authorization filter in order to execute API calls using ServicePrincipal credentials

  • 4680: Secrets are not deleted using user credentials, but using Service Principal Managed Identity

  • 4661: Check service-principal/managed-identity should call ServicePrincipal scim API to verify if it's present in workspace, not list jobs.

  • support multiple Security Architecture Options

Authorization Rules

  • 4395: Clear Cached Access Rules button

  • 4496: Add support for ActiveDirectory App Registration roles

  • 4659: Add an executive-role in LHM app

    • executive role can access all available workspaces

  • 4702: make Clear Cache function available to Users not only Admins

  • support Authorization Rules

  • support Public Subscriptions and Workspaces

🏎️ Performance Optimizations

  • 4386: Provide graceful shutdown for Consumption Loading

  • 4433: Refactor queue notification monitoring functionality in order to allow the monitoring of several jobs/pipelines in real time, without being limited by the size of the thread pool

  • 4561: Reduced prefilter max length due to the previous limit being too generous

  • 4685: Analyzer no longer checks if new metrics have been written

🎁 UX Optimizations

  • 4355: Authentication error not displayed in a friendly-mode

  • 4258: Persist Date Range Filter as query params in the URL

    • &startDate=2023-01-12&endDate=2023-01-18

    • facilitates specific job run sharing

https://bplm-demo.westus.cloudapp.azure.com/job-runs/424988970744905?subscriptionId=a63c1e51-40ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7883487973999049.9.azuredatabricks.net&startDate=2023-01-12&endDate=2023-01-18

❤️ Health Alerts

  • 4494: cost alerts api

  • 4574: Cost and Performance tabs

  • 4597: when I click "open in reports" from health alerts, confiture the workspace in the reports detailed view

⚙️ Operational Management – Telemetry & Consumption

  • 4253: Store job run analysis for all runs available in Databricks

  • 3769: Repair telemetry for jobs, notebooks and pipelines

  • 4462: Make 'Scheduled Analysis Intervals' section collapsable

  • 4568: Change the default number of Spark cores to be N - 2 for Telemetry Analysis engine

  • 4575: Remove Non Spark Workloads tab on Analysis History

  • 4590: Save signed-in user as "started by" info when triggering Run Now consumption jobs

  • 4590: Add startedBy field on consumption loading runs history

AWS support

  • use Dynamo DB instead of S3 as cloud storage solution for improved performance

  • use Azure AD Authentication provider in addition to Databricks identity provider with accounts API (username and password)

🎛️ Configuration

  • 4431: Add filter for "monitored" / "not monitored" when enabling or disabling the LHM agent

🖥️ Deployment

  • PR 2470: Marketplace readiness for v.2.0.0

    • Changed how ssh key is generated and handled on VM to accommodate marketplace deployment

    • Re-enabled paramsoutput script to provide quick and easy argument list for Configure-lakehouseAD.ps1

    • Various changes to pass arm ttk "best practices"

  • 4550: Add API to verify if service principal is enabled for background processors

  • 4592: Retry mechanism for powershell deployment

  • 4670: Scripts to set up LHM on existing infrastructure
    adding install scripts

  • 4717: Added global init scripts templates to enable monitor

🛠️ Miscellaneous

  • [4450] - Additional logic to resolve "unknown" workspaces

🐞Fixed Bugs

  • Bug 4449: Empty cost data in Cost Tenant Subscription env

  • Bug 4458: Autoscaling segment too short for streaming jobs spanning multiple days

    • Removed the databricks based adjustments of start time, end time.

  • Bug 4455: Decrease size of background login image

  • Bug 4456: Unnecessary config request is made

  • Bug 4451: Overview - entity label spans 2 rows

  • Bug 4461: Miscalculate cluster termination status by cluster events

  • Bug 4465: Fix analysis task priorities

  • Bug 4388: Enable monitoring on all jobs warning - columns move when the page is changed

  • Bug 4389: Enable monitoring on all jobs warning - notification window changes size when the page is changed

  • Bug 4457: Sorting by index of VM cost distribution does not work

  • Bug 4467: Fix cpu & memory metrics for in progress job runs

  • Bug 4444: Incorrect cost reporting for pipeline updates that run over multiple days

  • Bug 4492: In progress run not displayed on job runs

  • Bug 4502: I click on the STOP button and nothing happens for consumption loading / run history

  • Bug 4511: Fix consumption loading steps status calculation

  • Bug 4485: Jobs with sql tasks display incorrect provisioning status

  • Bug 4532: Overview - actual data bars are not visible

  • Bug 4534: java.lang.ClassCastException thrown when repairing telemetry

  • Bug 4346: Task metrics of a notebook ran on a single node cluster don't get analyzed

  • Bug 4544: Fix cluster idleness cost start_time computation

    • Truncate cluster idleness start_time and end_time granularity to seconds

    • Consumption loading fail caused by cluster idleness cost primary-key error

  • Bug 4537: Duplicate job and pipeline analysis errors on demo

  • Bug 4542: Create/link secret scope should not enable “Already exists”

  • Bug 4543: Service Principal enabled should not say optional

  • Bug 4562: Fix for correct avg/max/total duration of the job runs

    • inconsistency in the reported duration for jobs

  • Bug 4570: Violation of PRIMARY KEY constraint on notebook analysis

  • Bug 4576: NPE exception when repairing workspace telemetry

  • Bug 457: Fix cloudshell deployment scenario

  • Bug 4586: Job reports page changes build - Sorting by name does not work

  • Bug 4591: Repair telemetry throws exceptions for SKIPPED runs

  • Bug 3747: Notebooks runs distribution is duplicated in case REDACTED_LOCAL_PART is present in the name of the notebook

  • Bug 4548: overview totals for December should be the same across the different classification, Provider, Resource or Service

  • Bug 4631: add missing colon after Jobs label

  • Bug 4632: Hide border in the tooltip when no selection

  • Bug 4621: Barcharts not proportional with cost

  • Bug 4636: [overview] When resizing the browser window, the X axis values on the graph extend outside the container

  • Bug 4650: Fix pagination for reports

  • Bug 4554: 403 Forbidden error when getting job details as regular user

  • Bug 4637: Missing empty validation for date range on runs view

  • Bug 4635: Update cache endpoint called on reports page even if the user didn't change the workspace

  • Bug 4542: Create/link secret scope should have “Already exists” checkbox as default to false

  • Bug 4236: Telemetry analysis details/Cluster idleness - date incorrectly displayed

  • Bug 4666: When going back from all purpose clusters detailed view to overview no get requests are made

  • Bug 4665: Pipeline updates reporting is not authorized

Last commit id: 5262546d3

  • No labels