🎉 Highlights of 2.1 release
“Update Telemetry Now” - analyze on demand workflows that are in progress
Track SQL Warehouse costs per workspace
Health Alerts page
Improved scalability, performance and usability of Reports page when handling thousands of entities
🧭 Overview
4504: update styling for overview page
4706: workspace breakdown only aggregates on authorized workspaces when selecting 'All Workspaces'
4569: show workspace id when hovering on the name of workspace on overview page in the costs breakdown widget
4561: replace overview's cost telemetry alerts with Azure vs. Databricks Cost
4525: track SQL Warehouse costs per workspace
4682: update tooltip information on the cost breakdown widget to include information regarding SQL costs
4566: compute total workspace cost by service with estimated costs
Cost data per workspace includes all entities present in a workspace, i.e. Workflows, All-purpose clusters, VM pools, Delta Live Tables, SQL Warehouses.
Expenses incurred by resources drawn from a pool are only considered proportionally to the amount of use in Job clusters or All-purpose clusters.
Total workspace cost is represented by cloud infrastructure costs (e.g. VMs, storage, network) and Databricks costs.
4655: add highlight for current month cost on the subscription budget widget on the overview page
4654: improve visuals on selectable rows on the cost breakdown widget in the overview page
4653: make the widgets equal in width on the overview page
💾 Documentation
4638: add (i) icon next to Total Cost label to inform user about truncation
4642: update tooltips on settings page
4644: update info text on Reanalyze telemetry
Security Architecture Options
Authorization Rules
Public Subscriptions and Workspaces
📋 Reports
4623: use primary - apply and secondary - refresh for filters on redesigned Reports page
4624: use primary - apply and secondary - refresh for filters on redesigned Reports // job runs details page
4427: add cost & telemetry aggregations on Job overview with possibility to select the aggregated view
4415: Hide shuffle size column on Reports jobs overview by default
4501: Hide Sort by section on job runs
4384: add results search on Job Runs
as a user you can search for a particular word or id in the results displayed on the screen
4601: have TOTAL_COST as default sort by filter
4612: Add in progress icon with tooltip on new reporting view
4566: Compute total workspace cost by service with estimated costs. Fix unusedPoolsCost displaying N/A
4664: Compute idleness also for clusters for which we don't collect any task metrics
4688: Column tooltips on new reporting view for headers
On Demand Analysis
3611: Trigger job run analysis (background service) that analyzes the run which is in progress
click a button to trigger telemetry analyzer on this job in order to get the telemetry data already available aggregated and prepared for reporting
3901: Trigger only once at a time the "update telemetry" button per job run
4745: Show update now button for pipelines that have SETTING_UP_TABLES or RUNNING status
4622: hide the X on the sortBy filter until I hover over it
🔑 Security
4448: Google security warning displayed. Fixed double redirect parameter issue
4497: Support ServicePrincipal authentication to StorageAccount and Queue
4573: Improve logic for checking if service principal is mandatory
4652: ServicePrincipal Authorization filter in order to execute API calls using ServicePrincipal credentials
4680: Secrets are not deleted using user credentials, but using Service Principal Managed Identity
4661: Check service-principal/managed-identity should call ServicePrincipal scim API to verify if it's present in workspace, not list jobs.
support multiple Security Architecture Options
Authorization Rules
4395: Clear Cached Access Rules button
Lakehouse Monitor caches for a limited amount of time the access rules for the logged-in user that are configured in Databricks in order to improve responsiveness. If you recently changed your access rules in Databricks, please clear the cached access rules. Lakehouse Monitor will refresh the access rules of the logged in user.
4496: Add support for ActiveDirectory App Registration roles
4659: Add an executive-role in LHM app
executive role can access all available workspaces
4702: make Clear Cache function available to Users not only Admins
support Authorization Rules
support Public Subscriptions and Workspaces
4749: add authorization and caching on reports/clusters for signed-in user
🏎️ Performance Optimizations
4386: Provide graceful shutdown for Consumption Loading
4433: Refactor queue notification monitoring functionality in order to allow the monitoring of several jobs/pipelines in real time, without being limited by the size of the thread pool
4561: Reduced prefilter max length due to the previous limit being too generous
4685: Analyzer no longer checks if new metrics have been written
4541: Pagination for Workflows details (job runs)
🎁 UX Optimizations
4355: Authentication error not displayed in a friendly-mode
4673: Increase fonts by at least 1px for names and column headers on Reports page
4614: update navigation icons on the left bar
4463: update badges for Workflows (jobs) for monitor status
4258: Persist Date Range Filter as query params in the URL
&startDate=2023-01-12&endDate=2023-01-18facilitates specific job run sharing
https://bplm-demo.westus.cloudapp.azure.com/job-runs/424988970744905?subscriptionId=a63c1e51-40ae-4a34-b230-bf80e132c05c&workspaceHost=adb-7883487973999049.9.azuredatabricks.net&startDate=2023-01-12&endDate=2023-01-18
Bug 4721: workspace stats labels do not align properly and do not have proportional font size compared to the title tab
Bug 4722: improve color for workspace id on light theme on the cost breakdown on the overview page
Bug 4683: tooltip is outside of the screen if I navigate to the bottom right corner of the current month on the workspace stats widget in the overview page
Bug 4723: remove red color from the tooltip
❤️ Health Alerts
4494: cost alerts api
4574: Cost and Performance tabs
4730: authorization layer for the health and cost alerts
4597: when I click "open in reports" from health alerts, confiture the workspace in the reports detailed view
⚙️ Operational Management – Telemetry & Consumption
4762: Analyzer have to use the subscriptions and workspaces to which the SP has access.
4253: Store job run analysis for all runs available in Databricks
3769: Repair telemetry for jobs, notebooks and pipelines
4462: Make 'Scheduled Analysis Intervals' section collapsable
4568: Change the default number of Spark cores to be N - 2 for Telemetry Analysis engine
4575: Remove Non Spark Workloads tab on Analysis History
4590: Save signed-in user as "started by" info when triggering Run Now consumption jobs
4590: Add startedBy field on consumption loading runs history
AWS support
use Dynamo DB instead of S3 as cloud storage solution for improved performance
use Azure AD Authentication provider in addition to Databricks identity provider with accounts API (username and password)
🎛️ Configuration
4431: Add filter for "monitored" / "not monitored" when enabling or disabling the LHM agent
4770: add support to enable collector using global init scripts from app
🖥️ Deployment
PR 2470: Marketplace readiness for v.2.0.0
Changed how ssh key is generated and handled on VM to accommodate marketplace deployment
Re-enabled paramsoutput script to provide quick and easy argument list for Configure-lakehouseAD.ps1
Various changes to pass arm ttk "best practices"
4550: Add API to verify if service principal is enabled for background processors
4592: Retry mechanism for powershell deployment
4670: Scripts to set up LHM on existing infrastructure
adding install scripts4717: Added global init scripts templates to enable monitor
🛠️ Miscellaneous
[4450] - Additional logic to resolve "unknown" workspaces
[4704] - Add endpoints to list clusters/jobs/pipelines using subscriptionId&workspaceId instead of workspaceResourceId
🐞Fixed Bugs
Bug 4449: Empty cost data in Cost Tenant Subscription env
Bug 4458: Autoscaling segment too short for streaming jobs spanning multiple days
Removed the databricks based adjustments of start time, end time.
Bug 4455: Decrease size of background login image
Bug 4456: Unnecessary config request is made
Bug 4451: Overview - entity label spans 2 rows
Bug 4461: Miscalculate cluster termination status by cluster events
Bug 4465: Fix analysis task priorities
Bug 4388: Enable monitoring on all jobs warning - columns move when the page is changed
Bug 4389: Enable monitoring on all jobs warning - notification window changes size when the page is changed
Bug 4457: Sorting by index of VM cost distribution does not work
Bug 4467: Fix cpu & memory metrics for in progress job runs
Bug 4444: Incorrect cost reporting for pipeline updates that run over multiple days
Bug 4492: In progress run not displayed on job runs
Bug 4502: I click on the STOP button and nothing happens for consumption loading / run history
Bug 4511: Fix consumption loading steps status calculation
Bug 4485: Jobs with sql tasks display incorrect provisioning status
Bug 4532: Overview - actual data bars are not visible
Bug 4534: java.lang.ClassCastException thrown when repairing telemetry
Bug 4346: Task metrics of a notebook ran on a single node cluster don't get analyzed
Bug 4544: Fix cluster idleness cost start_time computation
Truncate cluster idleness
start_timeandend_timegranularity to secondsConsumption loading fail caused by cluster idleness cost primary-key error
Bug 4537: Duplicate job and pipeline analysis errors on demo
Bug 4542: Create/link secret scope should not enable “Already exists”
Bug 4543: Service Principal enabled should not say optional
Bug 4562: Fix for correct avg/max/total duration of the job runs
inconsistency in the reported duration for jobs
Bug 4570: Violation of PRIMARY KEY constraint on notebook analysis
Bug 4576: NPE exception when repairing workspace telemetry
Bug 457: Fix cloudshell deployment scenario
Bug 4586: Job reports page changes build - Sorting by name does not work
Bug 4591: Repair telemetry throws exceptions for SKIPPED runs
Bug 3747: Notebooks runs distribution is duplicated in case REDACTED_LOCAL_PART is present in the name of the notebook
Bug 4548: overview totals for December should be the same across the different classification, Provider, Resource or Service
Bug 4631: add missing colon after Jobs label
Bug 4632: Hide border in the tooltip when no selection
Bug 4621: Barcharts not proportional with cost
Bug 4636: [overview] When resizing the browser window, the X axis values on the graph extend outside the container
Bug 4650: Fix pagination for reports
Bug 4554: 403 Forbidden error when getting job details as regular user
Bug 4637: Missing empty validation for date range on runs view
Bug 4635: Update cache endpoint called on reports page even if the user didn't change the workspace
Bug 4542: Create/link secret scope should have “Already exists” checkbox as default to false
Bug 4236: Telemetry analysis details/Cluster idleness - date incorrectly displayed
Bug 4666: When going back from all purpose clusters detailed view to overview no get requests are made
Bug 4665: Pipeline updates reporting is not authorized
Bug 4727: Cluster idleness is not computed correctly
Bug 4731: NPE when managedIdentityAppId not configured and we run with service principal as background processors
Bug 4583: Secret scope button disabled for no reason
Bug 4738: Nested runs are not displayed as such
Bug 4734: Search is enabled when no workspace is selected
Bug 4584: Enable monitor on all jobs display empty list with jobs that cannot be monitorized
Bug 4633: consumption run history: switching page should scroll up and when reopening it it should be on page 1 again
Bug 4718: Skew Analysis Cannot Scroll
Bug 4743: Always open the run history on its first page
Bug 4403: Enable monitoring on all purpose-clusters seems successful when clusters are not actually monitored
Bug 4634: Filters bar scrolls off the screen while report view cuts off and has it's own scroll bar
Bug 4742: Change the error message displayed when the user is not authorized to access the runs of a particular job
Last commit id: cbd8ebe1d
0 Comments