aws-cdk-lib/aws-dynamodb: replicationRegions generate lots of AccessDenied errors in CloudTrail log

[cartmanez]

Describe the bug

When deploying, updating dynamodb table with replica regions, I observe access denied errors in cloud trail log

Expected Behavior

No access denied errors in cloudtrail log

Current Behavior

Bunch of AccessDenied errors generated during creation of a global table, also when adding/removing replica region, or removing whole table.

There are no errors in lambda logs or cloudformation stack events. Table construct works well with global tables.

However, these access denied errors trigger alarms we have for CIS control 3.1

Reproduction Steps

1. Deploy the following stack:

class DynamoDbReplicaAccessDeniedDemo extends Stack {

 constructor(scope: Construct, id: string, props?: StackProps) {

        super(scope, id, props);

        new Table(this, 'DemoTable', {
            partitionKey: {
                name: 'keyA',
                type: AttributeType.STRING
            },
            replicationRegions: ['us-west-1', 'us-west-2', 'us-east-2', 'eu-central-1', 'eu-west-1', 'eu-west-2', 'eu-west-3', 'ap-southeast-1', 'ap-southeast-2', ]
        })
    }
}

new DynamoDbReplicaAccessDeniedDemo(app, 'ddb-replica-access-denied-demo', {
    env: {
        region: 'us-east-1'
    }
});

2. Wait 15 minutes
3. Query CloudTrail log group with CloudWatch Logs Insights:

fields eventTime, awsRegion, eventName, userIdentity.arn, errorCode, errorMessage, @message
| filter eventName='DescribeTable' and errorCode like /AccessDenied/
| sort eventTime desc
| limit 2000

4. Notice there are bunch of AccessDenied errors like

User: arn:aws:sts:::assumed-role/ddb-replica-access-denied-IsCompleteHandlerService-XXXXXXXXX/ddb-replica-access-denied-IsCompleteHandlerYYYYYY-ZZZZZZZZZZ is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-east-2::table/ddb-replica-access-denied-demo-DemoTableXXXXXXX-PPPPPPPPPP because no identity-based policy allows the dynamodb:DescribeTable action

coming from all replica regions

Possible Solution

onEventHandler has dynamodb permissions in replica regions, while isCompleteHandler does not:

aws-cdk/packages/@aws-cdk/aws-dynamodb/lib/table.ts

Line 1664 in 433a1fa

onEventHandlerPolicy.grantPrincipal.addToPrincipalPolicy(new iam.PolicyStatement({

I suppose that the assumption was that calling DescribeTable in a region X will require only permissions in region X. However, it looks like AWS under the hood invokes DescribeTable in every replica region for global tables.

Additional Information/Context

No response

CDK CLI Version

2.38.1 (build a5ced21)

Framework Version

No response

Node.js Version

v14.20.0

OS

GNU/Linux

Language

Typescript

Language Version

4.7.4

Other information

No response

[mascur]

Hi @cartmanez,

Thank you for reporting this issue. I deployed a Stack using the code you provided and wasn't able to reproduce the issue you are having. Is there any additional information that you could provide that might help reproduce this issue?

[cartmanez]

Hi @mascur, thank you for looking into this.

Is you trail a multi-regional one? Access denied errors are happening in replica regions, not in the region where you deploy the stack.

Also, did you wait for 15 minutes or so before checking cloudtrail logs? They are usually delivered to log group with some delay

[rix0rrr]

This issue was for the existing Table construct, which used custom resources to implement table replication. We no longer recommend the use of the Table construct.

Instead, the TableV2 construct has been released in 2.95.1 (#27023) which maps to the AWS::DynamoDB::GlobalTable resource, has better support for replication and does not suffer from the issue described here.

---

Be aware that there are additional deployment steps involved in a migration from Table to TableV2. You need to do a RETAIN deployment, a delete deployment, then change the code to use TableV2 and then use cdk import. A link to a full guide will be posted once it is available.

Here are some other resources to get you started (using CfnGlobalTable instead of TableV2) if you want to get going on the migration:

• Migrating from Table to CfnGlobalTable, by Samaneh Utter, an AWS Senior Solutions Architect
• AWS CDK and Amazon DynamoDB Global Tables, by Wojciech Matuszewski from Stedi
• Migrating from Table to CfnGlobalTable, by Sharon Diskin from CloudBit