DynamoDB and SQS

Both the LHO Application and the LHO Agent running in the Databricks workspaces require access to DynamoDB tables and SQS queue that are created in the same AWS account as the LHO application, we will call this the “LHO_App_AWS_Account_Id” in the permission policies below:

LHO Agent permission policy

# LHO Agent IAM Role in the application-host AWS account (LHO_App_AWS_Account_Id)
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WriteToDynamoDbAndSqs",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchWriteItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "sqs:SendMessage"
            ],
            "Resource": [
                "arn:aws:sqs:<optional_region_or_*>:<LHO_App_AWS_Account_Id>:bplm*",
                "arn:aws:dynamodb:<optional_region_or_*>:<LHO_App_AWS_Account_Id>:table/bplm*"
            ]
        }
    ]
}

LHO Application permission policy

# LHO Application (VM) IAM Role in the application-host AWS account:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DynamoAndSQS",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "dynamodb:CreateTable",
                "dynamodb:UpdateTable",
                "sqs:GetQueueUrl",
                "dynamodb:UpdateTimeToLive",
                "dynamodb:BatchGetItem",
                "dynamodb:DescribeTable",
                "sqs:ReceiveMessage",
                "dynamodb:Scan",
                "dynamodb:Query",
                "sqs:CreateQueue"
            ],
            "Resource": [
                "arn:aws:sqs:<optional_source_region_or_*>:<LHO_App_AWS_Account_Id>:bplm*",
                "arn:aws:dynamodb:<optional_source_region_or_*>:<LHO_App_AWS_Account_Id>:table/bplm*"
            ]
        }
    ]
}

Trust policy for the LHO Agent IAM Role in the application-host AWS account:

This policy covers the scenario in which DynamoDb tables and SQS queue are accessed by LHO agents running on multiple AWS accounts hosting Dbx workspaces:

# LHO Agent IAM Role fully trusts a list of AWS accounts
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                  "arn:aws:iam::<Dbx_Workspace_AWS_Account1_id>:root",
                  "arn:aws:iam::<Dbx_Workspace_AWS_Account2_id>:root",
                  ...
                ]          
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

or

# LHO Agent IAM Role trusts a Databricks workspace Instance Profile IAM Role 
# in a particular AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<Dbx_Workspace_AWS_Account>:role/<Dbx_Wksp_Instance_Profile_IAM_Role>"
            },
            "Action": "sts:AssumeRole",
        }
    ]
}

Permission policy for the Databricks Workspace Instance Profile Roles that will assume the LHO_Agent_IAM_Role:

(see more info here: instance profile)

And in the Lakehouse Optimizer .env config file provide the AWS account ID and IAM Role name for the LHO agent that will be shipped to each monitored Databricks workspace:

Note that the AWS account ID of the LHO Agent, DynamoDB, SQS and LHO Application (VM) has the same value of LHO_App_Host_AWS_Account_ID