TABLE OF CONTENTS

• Installation scripts

• Release summary

• Complete release notes

  • Installation process

  • Authentication

  • Authorization

  • Connectors

  • Elasticsearch

  • MySQL

  • Azure / MS SQL

  • Materialization / Parquet store

  • Query Editor

  • Data Catalog

  • Service Management  

  • Miscellaneous

  • Logging information

  • Notifications

Installation scripts

On premise (click to download)

• conduit-2.5.5.tar.gz

• conduit-transformless-install-onprem.sh

Release summary

We are excited to announce the release of Conduit 2.5.5 version.

This release focuses on Materialization by adding new features, bug fixes and performance improvements.

Here are some highlights:

• caching has been renamed to Materialization.

• we introduced "materialize api" which enables you to start a materialization job on demand via a REST API call. See more in the Swagger documentation.

Complete release notes

Installation process

[CONDUITV3-1550] - provision data source for Grafana dashboards from the deployment script (#2520) (commit: 500b8eb)  

[CONDUITV3-1796] - Update script preserves SPARK_SCHEDULER_MODE variable (#2540)

[CONDUITV3-1702] - [UI] Updating info from credentials folder (#2538)

Authentication

[CONDUITV3-1677] - [UI] - Edit user changes are showed as saved even if update modal is not saved (#2489) (commit: c17151b)  

[CONDUITV3-101]  - Updating user password or deleting an internal user is not reflected in Conduit caches (#2471) (commit: 938eddf)  

[CONDUITV3-1679] - [UI][UsersRegistration] add configurable option to send email to all admins when new user registered

[CONDUITV3-1306] - protect SparkUI page with user/password - admins only allowed (#2536)

Authorization

[CONDUITV3-1696] - [UI][UX] rename label "User Admin" to "Users Administration" on the "Secure" section in Conduit Console Management (#2498) (commit: 8d1902a)  

[CONDUITV3-1576] - [UserRegistration] add user registration history (#2515) (commit: d3d00b9)

• user registration history is available in the metadata table saved by Conduit.

[CONDUITV3-1681] - [UI][UserRegistration] add configurable email for Conduit Support

[CONDUITV3-1819] - fix Save button on User settings does not work (#2559)

Connectors

[CONDUITV3-1510] - [PBI]Error when we have "-" in table name for PBI (#2486) (commit: abf71e5)  

[CONDUITV3-1684] - [Materialization] - Clone connector gets "materialize now" and "auto refresh" settings from original connector (#2496) (commit: 211c8d5)  

[CONDUITV3-1640] - [UI] remove partition column when connector updated the name of a column. (#2425) (commit: aee010f)  

[CONDUITV3-1651] - [UI] add (?) info for the read partition column on advanced step when editing a connector (#2493) (commit: 70263f1)

[CONDUITV3-1739] - [UX] update tooltip on connector edit advanced section for partitioning columns on write (#2511) (commit: 834fcb8)  

[CONDUITV3-1731] - [UX] update connector edit advanced tab (step 6) (#2508) (commit: 79ce02b)  

[CONDUITV3-1786] - [UX] update connector edit view for advanced step (#2548)

[CONDUITV3-1823] - [UX] rename step 4 on connector edit from "Virtualization" to "Materialization" (#2553)

Elasticsearch

[CONDUITV3-1732] - [ES] partitioning by column on flattened tables fails on columns containing '_' in the name (#2552)

MySQL

[CONDUITV3-1603] - [Mysql] improve ArrowFlight type conversion and sample types with all_types tables (#2546)

[CONDUITV3-1675] - "Type of query is not supported" error and failure in registering table to spark for credit_approval dataset of MySQL connector (#2580)

Azure / MS SQL

[CONDUITV3-XYZ]  - MSSQL fix tinyint through hive (#2482) (commit: 38645b3)  

[CONDUITV3-1704] - [SqlParser] support for SQL "case when" not working properly (#2507) (commit: ff67ebf)  

[CONDUITV3-69]   - Azure SQL native queries: Text fields: values in double quotes are missing when queries include these values (commit: 8511828)  

Materialization / Parquet store

[CONDUITV3-1714] - [UI] handle silently 502 errors on parquet store auto-refresh. Also removed redundant http call for enabled data sources (#2501) (commit: 51f21a3)  

[CONDUITV3-1722] - [UX][ParquetStore] improve tooltips message for columns in Parquet Store (#2500) (commit: 756fac8)  

[CONDUITV3-1722] - Added table name to parquet store metrics (#2505) (commit: e2de0ea)  

[CONDUITV3-1729] - [ParquetStore][Server] add materialization job history (#2502) (commit: 58006c9)

• materialization job history is available as metadata only which can be accessed via the Query Editor.    

[CONDUITV3-1728] - [ParquetStore][Server] add notification settings on failed jobs (#2518) (commit: 6f094a9)  

• click on "settings" button in the ParquetStore page and enable email notifications for failed jobs

[CONDUITV3-1750] - Azure Gov Storage account support for Conduit Parquet Store (commit: 9330241)  

[CONDUITV3-XYZ]  - fix integer overflow issue for ParquetCacheStatus (commit: 1ad1f7b)  

[CONDUITV3-1727] - [ParquetStore][UI] add resilience settings on failed jobs (#2521) (commit: 6410e33)  

[CONDUITV3-1706] - Unable to delete cached Oracle connector while job is still running in parquet store (#2529) (commit: 72afd7c)  

[CONDUITV3-1760] - Re-materialization job with partitioning columns throws on get LowerAndUpperBounds but leaves Job in Running state (#2530) (commit: 15763a8)  

[CONDUITV3-1750] - Azure Gov Storage account support for Conduit Parquet Store (commit: 3b26527)  

[CONDUITV3-XYZ]  - Improve error handling for getLowerAndUpperBounds

[CONDUITV3-1813] - Fix Parquet job cancel behavior (#2571)

[CONDUITV3-1846] - Fix connection leak on parquet materialization with JDBC partitioning (#2572)

[CONDUITV3-1842] - Removal of virtual dataset while materialization job is in progress doesn't work poperly (#2568)

[CONDUITV3-XYZ]  - improve error handling for materialization cancellation (#2567)

[CONDUITV3-1813] - Support transaction cancel for native query and spark query when materializing with partitions on RDMS datasets (#2557)

[CONDUITV3-1872] - "Partition column not found in schema struct" error when partitioning by nested columns is used (#2592)

[CONDUITV3-1874] - Race condition on table cache expiration between parquet store deletion code and re-cache code on auto-refresh (#2594)

[CONDUITV3-1860] - Handle materialization edge-cases on bde-server restart while job in progress (#2588)

[CONDUITV3-393]  - Parquet partitioning: validation that column selection doesn't conflict with materialization creation (#2583)

[CONDUITV3-1865] - Materialization is not allowed for ES connector containing dataset with nested columns (#2585)

[CONDUITV3-1764] - [Server] expose the Parquet Store location configuration in the Mgmt Console (#2579)

[CONDUITV3-1835] - Add materialization API to list un-materialized datasets and trigger materialization (#2569)

[CONDUITV3-1880] - Change endpoint path from "/parquet-cache" to "/parquet-store" (#2597)

[CONDUIT-XYZ] - Add default filter for parquet store (#2596)

[CONDUITV3-1879] - [UI] rename "cache now" option in connector editing to "materialize now"

Query Editor

[CONDUITV3-1648] - [UI][QueryEditor] add help (?) to query editor (#2497) (commit: 02f5535)  

Data Catalog

[CONDUITV3-1689] - Discover Page - Selected Items:  List and Card view buttons are only working for the Landing Page. (#2544)

Service Management  

[CONDUITV3-1447] - [UI][ServiceManagement] improve design to provide collapsable widgets for each server (#2472) (commit: 8a7f798)  

[CONDUITV3-1746] - [UX][ServiceManagement] do not close the others panels when clicking on a VM (#2514) (commit: 58cdf69)  

[CONDUITV3-1683] - [UI][ServiceManagement] add disk, cpu, memory information for each VM (#2522) (commit: d2722e8)  

[CONDUITV3-1660] - [UI] set "service management" as the default page for settings.

[CONDUITV3-1655] - [UI] when a new release is available, the dialog in the mgmt console should take me straight to the upgrades tab not license

[CONDUITV3-1501] - [UI] improve ServiceManagement UI to add options for disk percentage warning and email notifications enabled

[CONDUITV3-1749] - [UI][ServiceManagement] add configurable option to send email to all admins when a Conduit service fails

[CONDUITV3-1810] - [ServiceManagement] improve empty disk notifications (#2555)

[CONDUITV3-1820] - fix scenario when Disk, cpu, memory information cannot be acquired (#2564)

[CONDUITV3-1864] - [ServiceManagement] reduce number of digits to max 2 for disk space (#2584)

Miscellaneous

[CONDUITV3-1738] - [STMP] exception not handled when approving users with no SMTP configuration (#2506) (commit: 419d952)  

[CONDUITV3-1582] - add cleanup policy for spark application history folder (#2509) (commit: 19b30ab)  

[CONDUITV3-1583] - remove older docker images for conduit on upgrade (#2510) (commit: 1361af6)  

[CONDUITV3-1797] - [Swagger] Ignore spaces from beginning and on the end of infos written in the fields from Swagger (#2560)

[CONDUITV3-1638] - add index template for audit log index (#2551)

Logging information

[CONDUITV3-XYZ]  - set from debug to info for critical log entries upon parquet job startup (commit: 76ceaef)  

[CONDUITV3-1144] - add debug logs for mssql queries (#2534)

[CONDUITV3-223]  - Logs could use capturing how long it took to create parquet materialization (#2582)

Notifications

[CONDUITV3-1506] - send warning email to all admins when one VM has an unhealthy service (#2487) (commit: e3eca51)  

[CONDUITV3-1678] - [UserRegistration] add configurable option to send email to all admins when new user registered (#2495) (commit: e8160eb)  

[CONDUITV3-XYZ]  - Fixed missing triggeredBy value in materialized job failed email notification (#2527) (commit: ce7442e)  

[CONDUITV3-1777] - [UX] use server provided messages for user registration process (#2528) (commit: 82932f1)  

[CONDUITV3-1850] - Update email for failed jobs to have a more user-friendly structure (#2576)

[CONDUITV3-1801] - add email notifications for Backup on failed backup (on click and on scheduled) (#2554)

[CONDUITV3-1789] - expose jdbc connection timeout parameters in bde-server env configuration (#2539)

[CONDUITV3-1802] - [UX] update materialization job notifications email template (#2556)