...
LHM Application Role - a IAM role that is assigned to the EC2 Instance (VM) where the Lakehouse Monitor is deployed, the role allows sts:AssumeRole permission for cross account access or just regular permission policies for resource access.
LHM Agent Role - a IAM role that will be assumed by the Databricks Workspace Instance Profile Roles enabled for the Databricks workloads monitored by LHM.
LHM Application host AWS Account - AWS account where BPLM app (VM) is deployed and where DynamoDB and SQS artifacts are also created.
Databricks Workspace AWS account - AWS accounts hosting Databricks workspaces
Databricks
...
The S3 bucket storing the billable usage (Databricks consumption data) requires an S3 bucket policy that will specify the scope of access for the LHM Application:
...
Full AWS organization
...
Full AWS Account where LHM App is hosted
...
Code Block # Full AWS organization # Bucket policy { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*", "Condition": { "StringEquals": { "aws:PrincipalOrgID": "<org_id>" } } }, { "Sid": "S3ListBucket", "Effect": "Allow", "Principal": "*", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::<bucket>", "Condition": { "StringEquals": { "aws:PrincipalOrgID": "<org_id>" }, "StringLike": { "s3:prefix": "<path_prefix>/*" } } } ] }
| Code Block |
|---|
# Full AWS Account where LHM App is hosted
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:root"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
}
]
} |
| Code Block |
|---|
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
}
]
} |
KMS key policy:
| Code Block |
|---|
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
... the default statement for local trusting ...
},
{
"Sid": "Allow use of the key to LHM App IAM Role ",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "kms:Decrypt",
"Resource": "*"
}
]
} |
LHM Application IAM Role permission policy:
| Code Block |
|---|
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": [
"<path_prefix>/*"
]
}
}
},
{
"Sid": "DecryptKMSbucket",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": "<ARN_OF_CUSTOM_KMS_KEY_IN_SAME_REGION_AS_BUCKET>"
}
]
} |
Configuring Lakehouse monitor to read from s3:
| Code Block |
|---|
CONSUMPTION_BILLABLE_USAGE_PATH=s3a://<bucket>/<path_prefix>/billable-usage/csv
STORAGE_AWS_S3_REGION=<bucket_region> |
b) AWS managed KMS keys
You need to create a IAM role in the same AWS account as the S3 bucket with the Databricks billable usage logs. This role requires the following permission policies with access to the S3 bucket and AWS managed KMS key.
...
| Code Block |
|---|
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
},
{
"Sid": "DecryptKMSbucket",
"Action": "kms:Decrypt",
"Effect": "Allow",
"Resource": "<ARN_OF_AWS_MANAGED_KMS_KEY_IN_SAME_REGION_AS_BUCKET>"
}
]
} |
Trusting policy for the S3 role (only trusting a remote role version, for account-id or PrincipalOrgId, see the examples above):
...
| Code Block |
|---|
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "sts:AssumeRole"
}
]
}
|
LHM Application IAM Role permission policy:
...
| Code Block |
|---|
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<s3_aws_account_id>:role/<s3_role_name>"
}
]
} |
Configuring Lakehouse monitor to read from s3:
...
costs
DynamoDB and SQS:
Both the LHM Application and the LHM Agent running in the Databricks workspaces require access to DynamoDB tables and SQS queue that are created in the same AWS account as the LHM application, we will call this the “LHM_App_AWS_Account_Id” in the permission policies below:
...