DynamoDB and SQS

Both the LHO Application and the LHO Agent running in the Databricks workspaces require access to DynamoDB tables and SQS queue that are created in the same AWS account as the LHO application, we will call this the “LHM_App_AWS_Account_Id” in the permission policies below:

LHO Agent policy

# LHO Agent IAM Role in the application-host AWS account
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WriteToDynamoDbAndSqs",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:sqs:<optional_region_or_*>:<LHM_App_AWS_Account_Id>:bplm*",
                "arn:aws:dynamodb:<optional_region_or_*>:<LHM_App_AWS_Account_Id>:table/bplm*"
            ]
        }
    ]
}

LHO Application policy

# LHO Application (VM) IAM Role in the application-host AWS account:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DynamoAndSQS",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "dynamodb:CreateTable",
                "dynamodb:UpdateTable",
                "sqs:GetQueueUrl",
                "dynamodb:UpdateTimeToLive",
                "dynamodb:BatchGetItem",
                "dynamodb:DescribeTable",
                "sqs:ReceiveMessage",
                "dynamodb:Scan",
                "dynamodb:Query",
                "sqs:CreateQueue"
            ],
            "Resource": [
                "arn:aws:sqs:<optional_source_region_or_*>:<LHM_App_AWS_Account_Id>:bplm*",
                "arn:aws:dynamodb:<optional_source_region_or_*>:<LHM_App_AWS_Account_Id>:table/bplm*"
            ]
        }
    ]
}

Trust policy for the LHO Agent IAM Role in the application-host AWS account:

This policy covers the scenario in which DynamoDb tables and SQS queue are accessed by LHO agents running on multiple AWS accounts hosting Dbx workspaces:

# LHM Agent IAM Role fully trusts a list of AWS accounts
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                  "arn:aws:iam::<Dbx_Workspace_AWS_Account1_id>:root",
                  "arn:aws:iam::<Dbx_Workspace_AWS_Account2_id>:root",
                  ...
                ]          
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

or

# LHO Agent IAM Role trusts a Databricks workspace Instance Profile IAM Role 
# in a particular AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<Dbx_Workspace_AWS_Account>:role/<Dbx_Wksp_Instance_Profile_IAM_Role>"
            },
            "Action": "sts:AssumeRole",
        }
    ]
}

Permission policy for the Databricks Workspace Instance Profile Roles that will assume the LHM_Agent_IAM_Role:

(see more info here: instance profile)

And in the Lakehouse Monitor .env config file provide the AWS account ID and IAM Role name for the LHO agent that will be shipped to each monitored Databricks workspace:

Note that the AWS account ID of the LHM Agent, DynamoDB, SQS and LHM Application (VM) has the same value of LHM_App_Host_AWS_Account_ID