Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 16 Next »

Step 1) Required Resources

Lakehouse Monitor requires the following resources to already be created:

AWS Resource Requirements

Step 2) Configuration Prerequisites

  • AWS Secrets Manager needs to be configured with the following secret key value pairs. Suggested name for the secret is ‘bplm-credentials’:

    • storage-access-key - AWS Access Key used for accessing Amazon DynamoDB and Amazong SQS by the telemetry agent

    • storage-secret-key - AWS Secret Key DynamoDB secret key

      • DynamoDB is the telemetry data store, access from the LHM services or telemetry agents in Databricks workspaces can be enabled either with the access key/secret key pair or via IAM Roles/Credentials and Instance Profiles, in which case the key pair above becomes optional

    • service-account-username - Databricks service account username

    • service-account-password - Databricks service account password

      • required for access to the Billable Usage Logs of Databricks Accounts API

    • mssql-password - SQL Login password for the SQL Database

    • application-encryption-secret - encryption key for storing PATs (Personal Access Tokens) and the Databricks Accounts credentials (billable usage logs) in the LHM SQL database

    • msft-provider-auth-secret - Client secret value from azure app registration

Step 3) Installation procedure

1. SSH into the BPLM VM configured at Step 1) Required Resources.

  • Download the install archive by running the following command:

    • Ubuntu:
      wget https://bplmdemoappstg.blob.core.windows.net/deployment/vm-aws/ubuntu.zip

    • CentOS:
      wget https://bplmdemoappstg.blob.core.windows.net/deployment/vm-aws/centos.zip

  • Extract the archive contents
    unzip ubuntu.zip / unzip centos.zip

In the destination directory you should see the following files:

  .env
  docker-compose.yml
  setup.sh
  start.sh

  1. Before you start setup you need to fill out the .env file with the required information). Open the file in your editor of choice and fill in the values.

    1. Please find a brief explanation of the .env values below

Lakehouse uses databricks authentication currently with an optional setup of Azure AD as the identity provider. If you will not be using AAD, you do not need to fill out SERVICE_PRINCIPAL_CLIENTID, SERVICE_PRINCIPAL_TENANTID,or SERVICE_PRINCIPAL_CLIENTSECRET

You can also remove “,active-directory" from AUTHENTICATION_PROVIDER

LOG_LEVEL=info
LOG_LEVEL_APP=info
LOG_LEVEL_HTTP_HEADERS=error

APPSERVICE_URL=<eg:https://demo.aws-bplm.com> 

SQL_DATABASE=<App database name>
SQL_SERVER_HOST=<eg:192.168.4.10 or endpoint url>
SQL_USER=<eg:sql_admin>

STORAGE_AWS_REGION=<eg:us-west-1>
STORAGE_AWS_TABLE_PREFIX=bplm

AWS_SECRETS_MANAGER_ENABLED=true
AWS_SECRETS_MANAGER_REGION=<eg:us-west-1>
BPLM_SECRET_NAME=<name of secret created>

SERVER_SSL_ENABLED=true
SERVER_SSL_KEY-STORE=/keystore/bplm.p12
SERVER_SSL_KEY-STORE-PASSWORD=
SERVER_SSL_KEY-STORE-TYPE=PKCS12
SERVER_SSL_KEY-ALIAS=bplm
SERVER_SSL_KEY-PASSWORD=

SERVICE_PRINCIPAL_CLIENTID=<eg: 925accb1-8506-4ec4-a90b-b1b0e6d8a5eb>
SERVICE_PRINCIPAL_TENANTID=<eg: 03786a4c-412b-4fac-a981-b4c5bcbc55b7>
SERVICE_PRINCIPAL_CLIENTSECRET=${msft-provider-auth-secret}

DATABRICKS_ACCOUNT_ID=<eg: 56293882-89e7-4ecd-a5f7-cb61e68a54f0>
DATARICKS_SERVICE_PRINCIPAL=<eg: 48de6ad6-ff14-403d-b842-d4ce5da4662f>
ACTIVE-DIRECTORY_HOST=https://login.microsoftonline.com
ACTIVE-DIRECTORY_TOKEN-ENDPOINT=/oauth2/v2.0/token
ACTIVE-DIRECTORY_AUTHORIZE-ENDPOINT=/oauth2/v2.0/authorize
ACTIVE-DIRECTORY_JWK-ENDPOINT=/discovery/keys
ACTIVE-DIRECTORY_USER-INFO-URI=https://graph.microsoft.com/oidc/userinfo

CLOUD_PROVIDER=AWS
AUTHENTICATION_PROVIDER=databricks-account,active-directory
SPRING_PROFILES_ACTIVE=production-aws
SERVER_SERVLET_SESSION_PERSISTENT=true
SERVER_SERVLET_SESSION_STORE_DIR=<eg: /home/localuser/dockerless-env/spring-session/session>
ADMIN_APP_ROLE=internal_user
METRIC_PROCESSING_ENABLED=false
STORAGE_THROUGH_IAM_CREDENTIALS=true
APPLICATION_NOTIFICATION_JOBNOTIFICATIONQUEUENAME=bplm
#metric.queueMonitoring.compactionTimeout=PT25M

4. Run ./setup.sh providing the domain you wish to create an SSL cert for, the version of the lakehouse monitor, and an admin email that will be used to configure certbot’s notifications when creating an SSL certificate.

If you do not currently have a registered DNS entry for the lakehouse monitor, you can skip setting up SSL certs by not supplying the cert_domain or email_certbot arguments.

eg: ./setup.sh --cert_domain "lakehouse-monitor.company.com" --version 2.3 --email_certbot notifications@company.com

5. After the setup script completes, run start.sh to pull down the application container and start it

  • For being able to run the start script you will need to provide the Blueprint Docker Registry username and password that will be used by docker to pull the BPLM images from the Blueprint container registry: bplm-acr-token / <password to be provided upon deployment>

eg: ./start.sh example-acr-user someStrongPassword

  • No labels