Versions Compared

Version Old Version 17 New Version 18
Changes made by

Denis Pitul

Denis Pitul

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • AWS Secrets Manager needs to be configured with the following secret key value pairs. Suggested name for the secret is ‘bplm-credentials’:

    • storage-access-key - AWS Access Key used for accessing Amazon DynamoDB and Amazong Amazon SQS by the telemetry agent

    • storage-secret-key - AWS Secret Key DynamoDB secret key

      • DynamoDB is the telemetry data store, access from the LHM services or telemetry agents in Databricks workspaces can be enabled either with the access key/secret key pair or via IAM Roles/Credentials and Instance Profiles, in which case the key pair above becomes optional

    • service-account-username - Databricks service account username

    • service-account-password - Databricks service account password

      • required for access to the Billable Usage Logs of Databricks Accounts API

    • mssql-password - SQL Login password for the SQL Database

    • application-encryption-secret - encryption key for storing PATs (Personal Access Tokens) and the Databricks Accounts credentials (billable usage logs) in the LHM SQL database

    • msft-provider-auth-secret - Client secret value from azure app registration

  • Note: the storage-access-key and storage-secret-key are optional. They become required only if you choose to NOT use the IAM Role for accessing DynamoDB and SQS.

Step 3) Installation procedure

...

Info

Lakehouse uses databricks authentication currently with an optional setup of Azure AD as the identity provider. If you will not be using AAD, you do not need to fill out SERVICE_PRINCIPAL_CLIENTID, SERVICE_PRINCIPAL_TENANTID,or SERVICE_PRINCIPAL_CLIENTSECRET

You can also In this case you must remove “,active-directory" from AUTHENTICATION_PROVIDER

Code Block
LOG_LEVEL=info
LOG_LEVEL_APP=info
LOG_LEVEL_HTTP_HEADERS=error

APPSERVICE_URL=<eg:https://demo.aws-bplm.com>


SQL_DATABASE=<Appmaster
database name>
SQL_SERVER_HOST=<eg:192.168.4.10 or endpoint DNS url>name>
SQL_USER=<eg:sql_admin>

STORAGE_AWS_REGION=<eg:us-west-1>
STORAGE_AWS_TABLE_PREFIX=bplm

AWS_SECRETS_MANAGER_ENABLED=true
AWS_SECRETS_MANAGER_REGION=<eg:us-west-1>
BPLM_SECRET_NAME=<name of secretthe secrets created>manager secret>
SERVER_SSL_ENABLED=true
SERVER_SSL_KEY-STORE=/keystore/bplm.p12
SERVER_SSL_KEY-STORE-PASSWORD=
SERVER_SSL_KEY-STORE-TYPE=PKCS12
SERVER_SSL_KEY-ALIAS=bplm
SERVER_SSL_KEY-PASSWORD=

SERVICE_PRINCIPAL_CLIENTID=<eg: 925accb1-8506-4ec4-a90b-b1b0e6d8a5eb>
SERVICE_PRINCIPAL_TENANTID=<eg: 03786a4c-412b-4fac-a981-b4c5bcbc55b7>
SERVICE_PRINCIPAL_CLIENTSECRET=<secret value or ${msft-provider-auth-secret}secret key name from secrets manager}>

DATABRICKS_ACCOUNT_ID=<eg: 56293882-89e7-4ecd-a5f7-cb61e68a54f0>
DATARICKS_SERVICE_PRINCIPAL=<eg: 48de6ad6-ff14-403d-b842-d4ce5da4662f>
ACTIVE-DIRECTORY_HOST=https://login.microsoftonline.com
ACTIVE-DIRECTORY_TOKEN-ENDPOINT=/oauth2/v2.0/token
ACTIVE-DIRECTORY_AUTHORIZE-ENDPOINT=/oauth2/v2.0/authorize
ACTIVE-DIRECTORY_JWK-ENDPOINT=/discovery/keys
ACTIVE-DIRECTORY_USER-INFO-URI=https://graph.microsoft.com/oidc/userinfo

CLOUD_PROVIDER=AWS
AUTHENTICATION_PROVIDER=databricks-account,active-directory
SPRING_PROFILES_ACTIVE=production-aws
SERVER_SERVLET_SESSION_PERSISTENT=true
SERVER_SERVLET_SESSION_STORE_DIR=<eg: /home/localuser/dockerless-envubuntu/spring-session/session>session
ADMIN_APP_ROLE=internal_userbplm-admin
METRIC_PROCESSING_ENABLED=false
STORAGE_THROUGH_IAM_CREDENTIALS=true
#metric.queueMonitoring.compactionTimeout=PT25M
APPLICATION_NOTIFICATION_JOBNOTIFICATIONQUEUENAME=bplm
#metric.queueMonitoring.compactionTimeout=PT25M
<prefix for sqs names>

Note: due to the docker version provided by CentOS the SERVICE_PRINCIPAL_CLIENTSECRET can not be pulled from the secrets manager.

4. Run ./setup.sh providing the domain you wish to create an SSL cert for, the version of the lakehouse monitor, and an admin email that will be used to configure certbot’s notifications when creating an SSL certificate.

...