Log Delivery for Billable usages in S3
The S3 bucket storing the billable usage (Databricks consumption data) requires an S3 bucket policy that will specify the scope of access for the LHM Application:
Full AWS organization
Full AWS Account where LHM App is hosted
Exactly the IAM Role of the LHM Application in the AWS Account hosting it
Depending on client security configuration on S3 bucket, two options are available for cross-account access.
a) Bucket policy and custom KMS key policy: applicable when custom KMS keys are used. The custom key and the bucket must belong to the same AWS region.
Bucket policy:
# Full AWS organization # Bucket policy { "Version": "2012-10-17", "Statement": [ { "Sid": "S3ReadObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*", "Condition": { "StringEquals": { "aws:PrincipalOrgID": "<org_id>" } } }, { "Sid": "S3ListBucket", "Effect": "Allow", "Principal": "*", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::<bucket>", "Condition": { "StringEquals": { "aws:PrincipalOrgID": "<org_id>" }, "StringLike": { "s3:prefix": "<path_prefix>/*" } } } ] }
# Full AWS Account where LHM App is hosted
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:root"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
}
]
}
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
}
]
}
KMS key policy:
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
... the default statement for local trusting ...
},
{
"Sid": "Allow use of the key to LHM App IAM Role ",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
LHM Application IAM Role permission policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": [
"<path_prefix>/*"
]
}
}
},
{
"Sid": "DecryptKMSbucket",
"Action": [
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": "<ARN_OF_CUSTOM_KMS_KEY_IN_SAME_REGION_AS_BUCKET>"
}
]
}
Configuring Lakehouse monitor to read from s3:
CONSUMPTION_BILLABLE_USAGE_PATH=s3a://<bucket>/<path_prefix>/billable-usage/csv STORAGE_AWS_S3_REGION=<bucket_region>
b) AWS managed KMS keys
You need to create a IAM role in the same AWS account as the S3 bucket with the Databricks billable usage logs. This role requires the following permission policies with access to the S3 bucket and AWS managed KMS key.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "S3ReadObject",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket>/<path_prefix>/*"
},
{
"Sid": "S3ListBucket",
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket>",
"Condition": {
"StringLike": {
"s3:prefix": "<path_prefix>/*"
}
}
},
{
"Sid": "DecryptKMSbucket",
"Action": "kms:Decrypt",
"Effect": "Allow",
"Resource": "<ARN_OF_AWS_MANAGED_KMS_KEY_IN_SAME_REGION_AS_BUCKET>"
}
]
}
|
Trusting policy for the S3 role (only trusting a remote role version, for account-id or PrincipalOrgId, see the examples above):
# Exactly the IAM Role of the LHM Application in the AWS Account hosting it
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<LHM_App_Host_AWS_Account_Id>:role/<LHM_App_IAM_Role>"
},
"Action": "sts:AssumeRole"
}
]
}
|
LHM Application IAM Role permission policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::<s3_aws_account_id>:role/<s3_role_name>"
}
]
}
|
Configuring Lakehouse monitor to read from s3:
CONSUMPTION_BILLABLE_USAGE_PATH=s3a://<bucket>/<path_prefix>/billable-usage/csv STORAGE_AWS_S3_REGION=<bucket_region> CROSS_ACCOUNT_ASSUME_IAM_ROLE_S3_DBX_BILLING_APP=arn:aws:iam::<s3_aws_account_id>:role/<s3_dbx_billing_role_name> |
0 Comments