Cloud Costs via AWS CostExplorer

AWS CostExplorer

LHO Application IAM Role will assume an IAM Role in the Databricks Workspace AWS Account with a permission policy to access Cost Explorer data in that AWS Account Id:

IAM_Role_Cost_Explorer permission policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGetCostAndUsages",
            "Effect": "Allow",
            "Action": "ce:GetCostAndUsage",
            "Resource": "*"
        }
    ]
}

Trust policy for the IAM_Role_Cost_Explorer that allows the LHO Application IAM Role in the app-host AWS Account to assume the cost explorer role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<LHO_App_Host_AWS_Account_ID>:role/<LHO_App_IAM_Role>"
            },
            "Action": "sts:AssumeRole",
        }
    ]
}

Permissions for cost-tagging NAT and S3

In order to obtain workspace storage and network costs, the LHO should be provided permissions to tag S3 bucket and NAT Gateway associated with the Databricks workspace:

IAM_Role_Cost_Tag_For_S3_And_NAT permission policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BplmS3TaggingWkspStorageCostPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketTagging",
                "s3:PutBucketTagging"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "BplmNatGatewayTaggingPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteTags",
                "ec2:CreateTags"
            ],
            "Resource": [
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:natgateway/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:natgateway/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:vpc/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:internet-gateway/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:elastic-ip/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:route-table/*",
              "arn:aws:ec2:*:<Databricks_Wksp_AWS_Account_Id>:vpc-endpoint/*"
            ]
        },
        {
            "Sid": "BplmNatGatewayVpcsTaggingPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVpcs",
                "ec2:DescribeNatGateways",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeAddresses",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeRouteTables"
            ],
            "Resource": "*"
        }
    ]
}

Trust policy for the IAM_Role_Cost_Tag_For_S3_And_NAT that allows the LHO Application IAM Role in the app-host AWS Account to assume the role:

LHO application IAM Role permission policy:

And in the Lakehouse Optimizer .env config file provide the source role name for cost explorer and tags.

In case of multiple AWS accounts hosting cloud resources for Databricks workspaces, you need to provide a comma-separated list of roles, one role for each AWS account.