🎉 Highlights of 2.1.0 release
“Update Telemetry Now” - analyze on demand workflows that are in progress
Track SQL Warehouse costs per workspace
Health Alerts page
Improved scalability, performance and usability of Reports page when handling thousands of entities
🧭 Overview
4504: update styling for overview page
4569: show workspace id when hovering on the name of workspace on overview page in the costs breakdown widget
4561: replace overview's cost telemetry alerts with Azure vs. Databricks Cost
4525: track SQL Warehouse costs per workspace
4682: update tooltip information on the cost breakdown widget to include information regarding SQL costs
4566: compute total workspace cost by service with estimated costs
4655: add highlight for current month cost on the subscription budget widget on the overview page
4654: improve visuals on selectable rows on the cost breakdown widget in the overview page
4653: make the widgets equal in width on the overview page
💾 Documentation
4638: add (i) icon next to Total Cost label to inform user about truncation
4642: update tooltips on settings page
4644: update info text on Reanalyze telemetry
Security Architecture Options
Authorization Rules
Public Subscriptions and Workspaces
📋 Reports
4427: add cost & telemetry aggregations on Job overview with possibility to select the aggregated view
4415: Hide shuffle size column on Reports jobs overview by default
4501: Hide Sort by section on job runs
4384: add results search on Job Runs
4601: have TOTAL_COST as default sort by filter
4612: Add in progress icon with tooltip on new reporting view
4566: Compute total workspace cost by service with estimated costs. Fix unusedPoolsCost displaying N/A
4664: Compute idleness also for clusters for which we don't collect any task metrics
4688: Column tooltips on new reporting view for headers
On Demand Analysis
3611: Trigger job run analysis (background service) that analyzes the run which is in progress
3901: Trigger only once at a time the "update telemetry" button per job run
🔑 Security
4448: Google security warning displayed. Fixed double redirect parameter issue
4497: Support ServicePrincipal authentication to StorageAccount and Queue
4573: Improve logic for checking if service principal is mandatory
4652: ServicePrincipal Authorization filter in order to execute API calls using ServicePrincipal credentials
4680: Secrets are not deleted using user credentials, but using Service Principal Managed Identity
4661: Check service-principal/managed-identity should call ServicePrincipal scim API to verify if it's present in workspace, not list jobs.
support multiple Security Architecture Options
Authorization Rules
4395: Clear Cached Access Rules button
4496: Add support for ActiveDirectory App Registration roles
4659: Add an executive-role in LHM app
executive role can access all available workspaces
4702: make Clear Cache function available to Users not only Admins
support Authorization Rules
support Public Subscriptions and Workspaces
🏎️ Performance Optimizations
4386: Provide graceful shutdown for Consumption Loading
4433: Refactor queue notification monitoring functionality in order to allow the monitoring of several jobs/pipelines in real time, without being limited by the size of the thread pool
4561: Reduced prefilter max length due to the previous limit being too generous
4685: Analyzer no longer checks if new metrics have been written
🎁 UX Optimizations
4355: Authentication error not displayed in a friendly-mode
4258: Persist Date Range Filter as query params in the URL
&startDate=2023-01-12&endDate=2023-01-18facilitates specific job run sharing
https://bplm-demo.westus.cloudapp.azure.com/job-runs/424988970744905?subscriptionId=a63c1e51-40ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7883487973999049.9.azuredatabricks.net&startDate=2023-01-12&endDate=2023-01-18
❤️ Health Alerts
4494: cost alerts api
4574: Cost and Performance tabs
4597: when I click "open in reports" from health alerts, confiture the workspace in the reports detailed view
⚙️ Operational Management – Telemetry & Consumption
4253: Store job run analysis for all runs available in Databricks
3769: Repair telemetry for jobs, notebooks and pipelines
4462: Make 'Scheduled Analysis Intervals' section collapsable
4568: Change the default number of Spark cores to be N - 2 for Telemetry Analysis engine
4575: Remove Non Spark Workloads tab on Analysis History
4590: Save signed-in user as "started by" info when triggering Run Now consumption jobs
4590: Add startedBy field on consumption loading runs history
🎛️ Configuration
4431: Add filter for "monitored" / "not monitored" when enabling or disabling the LHM agent
🖥️ Deployment
PR 2470: Marketplace readiness for v.2.0.0
Changed how ssh key is generated and handled on VM to accommodate marketplace deployment
Re-enabled paramsoutput script to provide quick and easy argument list for Configure-lakehouseAD.ps1
Various changes to pass arm ttk "best practices"
4550: Add API to verify if service principal is enabled for background processors
4592: Retry mechanism for powershell deployment
4670: Scripts to set up LHM on existing infrastructure
adding install scripts4717: Added global init scripts templates to enable monitor
🛠️ Miscellaneous
[4450] - Additional logic to resolve "unknown" workspaces
🐞Fixed Bugs
Bug 4449: Empty cost data in Cost Tenant Subscription env
Bug 4458: Autoscaling segment too short for streaming jobs spanning multiple days
Removed the databricks based adjustments of start time, end time.
Bug 4455: Decrease size of background login image
Bug 4456: Unnecessary config request is made
Bug 4451: Overview - entity label spans 2 rows
Bug 4461: Miscalculate cluster termination status by cluster events
Bug 4465: Fix analysis task priorities
Bug 4388: Enable monitoring on all jobs warning - columns move when the page is changed
Bug 4389: Enable monitoring on all jobs warning - notification window changes size when the page is changed
Bug 4457: Sorting by index of VM cost distribution does not work
Bug 4467: Fix cpu & memory metrics for in progress job runs
Bug 4444: Incorrect cost reporting for pipeline updates that run over multiple days
Bug 4492: In progress run not displayed on job runs
Bug 4502: I click on the STOP button and nothing happens for consumption loading / run history
Bug 4511: Fix consumption loading steps status calculation
Bug 4485: Jobs with sql tasks display incorrect provisioning status
Bug 4532: Overview - actual data bars are not visible
Bug 4534: java.lang.ClassCastException thrown when repairing telemetry
Bug 4346: Task metrics of a notebook ran on a single node cluster don't get analyzed
Bug 4544: Fix cluster idleness cost start_time computation
Truncate cluster idleness
start_timeandend_timegranularity to secondsConsumption loading fail caused by cluster idleness cost primary-key error
Bug 4537: Duplicate job and pipeline analysis errors on demo
Bug 4542: Create/link secret scope should not enable “Already exists”
Bug 4543: Service Principal enabled should not say optional
Bug 4562: Fix for correct avg/max/total duration of the job runs
inconsistency in the reported duration for jobs
Bug 4570: Violation of PRIMARY KEY constraint on notebook analysis
Bug 4576: NPE exception when repairing workspace telemetry
Bug 457: Fix cloudshell deployment scenario
Bug 4586: Job reports page changes build - Sorting by name does not work
Bug 4591: Repair telemetry throws exceptions for SKIPPED runs
Bug 3747: Notebooks runs distribution is duplicated in case REDACTED_LOCAL_PART is present in the name of the notebook
Bug 4548: overview totals for December should be the same across the different classification, Provider, Resource or Service
Bug 4631: add missing colon after Jobs label
Bug 4632: Hide border in the tooltip when no selection
Bug 4621: Barcharts not proportional with cost
Bug 4636: [overview] When resizing the browser window, the X axis values on the graph extend outside the container
Bug 4650: Fix pagination for reports
Bug 4554: 403 Forbidden error when getting job details as regular user
Bug 4637: Missing empty validation for date range on runs view
Bug 4635: Update cache endpoint called on reports page even if the user didn't change the workspace
Bug 4542: Create/link secret scope should have “Already exists” checkbox as default to false
Bug 4236: Telemetry analysis details/Cluster idleness - date incorrectly displayed
Bug 4666: When going back from all purpose clusters detailed view to overview no get requests are made
Bug 4665: Pipeline updates reporting is not authorized
Last commit id: 5262546d3
0 Comments