Prerequisite: Follow instructions provided to setup lakehouse infrastructure resources
TODO: provide link to confluence page with detailed infrastructure requirements and setup
1. Fill out .env file with appropriate values from your setup
Lakehouse uses databricks authentication currently with an optional setup of Azure AD as the identity provider. If you will not be using AAD, you do not need to fill out SERVICE_PRINCIPAL_CLIENTID, SERVICE_PRINCIPAL_TENANTID,or SERVICE_PRINCIPAL_CLIENTSECRET
You can also remove “,active-directory" from AUTHENTICATION_PROVIDER
LOG_LEVEL=info LOG_LEVEL_APP=info LOG_LEVEL_HTTP_HEADERS=error APPSERVICE_URL=<eg:https://demo.aws-bplm.com> SQL_DATABASE=master SQL_SERVER_HOST=<eg:192.168.4.10> SQL_USER=<eg:sql_admin> STORAGE_AWS_REGION=<eg:us-west-1> STORAGE_AWS_TABLE_PREFIX=bplm AWS_SECRETS_MANAGER_ENABLED=true AWS_SECRETS_MANAGER_REGION=<eg:us-west-1> SERVER_SSL_ENABLED=true SERVER_SSL_KEY-STORE=/keystore/bplm.p12 SERVER_SSL_KEY-STORE-PASSWORD= SERVER_SSL_KEY-STORE-TYPE=PKCS12 SERVER_SSL_KEY-ALIAS=bplm SERVER_SSL_KEY-PASSWORD= SERVICE_PRINCIPAL_CLIENTID=<eg: 925accb1-8506-4ec4-a90b-b1b0e6d8a5eb> SERVICE_PRINCIPAL_TENANTID=<eg: 03786a4c-412b-4fac-a981-b4c5bcbc55b7> SERVICE_PRINCIPAL_CLIENTSECRET=<secret value> DATABRICKS_ACCOUNT_ID=<eg: 56293882-89e7-4ecd-a5f7-cb61e68a54f0> DATARICKS_SERVICE_PRINCIPAL=<eg: 48de6ad6-ff14-403d-b842-d4ce5da4662f> ACTIVE-DIRECTORY_HOST=https://login.microsoftonline.com ACTIVE-DIRECTORY_TOKEN-ENDPOINT=/oauth2/v2.0/token ACTIVE-DIRECTORY_AUTHORIZE-ENDPOINT=/oauth2/v2.0/authorize ACTIVE-DIRECTORY_JWK-ENDPOINT=/discovery/keys ACTIVE-DIRECTORY_USER-INFO-URI=https://graph.microsoft.com/oidc/userinfo CLOUD_PROVIDER=AWS AUTHENTICATION_PROVIDER=databricks-account,active-directory SPRING_PROFILES_ACTIVE=production-aws SERVER_SERVLET_SESSION_PERSISTENT=true SERVER_SERVLET_SESSION_STORE_DIR=<eg: /home/localuser/dockerless-env/spring-session/session> ADMIN_APP_ROLE=internal_user METRIC_PROCESSING_ENABLED=false #metric.queueMonitoring.compactionTimeout=PT25M
2. In the same region as the rest of the resources, create a secret in Secret Manager with the name 'bplm-credentials' and add the key value pairs below ( example given as plain text JSON entry )
{
"storage-access-key":"<>",
"storage-secret-key":"<>",
"service-account-username":"<databricks service account name>",
"service-account-password":"<>",
"mssql-password":"<sql admin password>",
"application-encryption-secret":"<>"
}
3. Copy setup scripts and completed .env file template onto the host VM
docker-compose.yml, .env, setup.sh, and start.sh
TODO: setup download for these files and provide link
4. Run ./setup.sh providing the domain you wish to create an SSL cert for, the version of the lakehouse monitor, and an admin email that will be used to configure certbot’s notifications when creating an SSL certificate.
If you do not currently have a registered DNS entry for the lakehouse monitor, you can skip setting up SSL certs by not supplying the cert_domain or email_certbot arguments.
eg: ./setup.sh --cert_domain "lakehouse-monitor.company.com" --version 1.4.0 --email_certbot notifications@company.com
5. After the setup script completes, run start.sh to pull down the application container and start it
eg: ./start.sh example-acr-user someStrongPassword
Add Comment