(secretsmanager): RotationSchedule resource creation race causes some stack deployments to fail

[gavllew]

Describe the bug

When using CDK to set up a Secrets Manager RotationSchedule, creation of the AWS::SecretsManager::RotationSchedule resource can fail (in a small proportion of deployments) with the following error:

Secrets Manager cannot invoke the specified Lambda function. Ensure that the function policy grants access to the principal secretsmanager.amazonaws.com. (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException

From looking at the Stack events, I can see that the Lambda Invoke permission and the RotationSchedule are being created at the same time, so it's down to luck whether the RotationSchedule's test invoke succeeds or fails, though in most cases it seems to succeed. An explicit dependency to ensure the Lambda resource policy is created first would improve the deployment reliability.

Expected Behavior

The deployment succeeds reliably.

Current Behavior

The deployment failed with the above error in 2 out of 18 deployments.

Reproduction Steps

Create a new typescript CDK project:

mkdir cdk-bug; cd cdk-bug
cdk init --language=typescript

Replace lib/cdk-bug-stack.ts with the following:

import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { Construct } from 'constructs';

export class CdkBugStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const secret = new Secret(this, "Secret", {});
    const rotationLambda = new lambda.Function(this, "RotationLambda", {
      runtime: lambda.Runtime.NODEJS_18_X,
      handler: 'index.handler',
      code: lambda.Code.fromInline('exports.handler = async () => { console.log("Fake rotation lambda"); };'),
    });

    secret.addRotationSchedule("RotationSchedule", {
      rotationLambda,
    });
  }
}

A cdk synth shows that the SecretRotationSchedule49AED07D resource has no dependency on the AWS::Lambda::Permission resource RotationLambdaInvokeN0a2GKfZP0JmDqDEVhhu6A0TUv3NyNbk4YMFKNc852E0E9A.

Possible Solution

Add an explicit dependency to ensure the Lambda resource policy is created before the RotationSchedule.

Additional Information/Context

No response

CDK CLI Version

2.88.0 (build 5d497f9)

Framework Version

2.87.0

Node.js Version

v18.17.0

OS

MacOS 13.4.1 (c) (22F770820d)

Language

Typescript

Language Version

TypeScript (5.1.6)

Other information

No response

[peterwoodworth]

Thanks, this makes sense. We add the permission here in the code, but there isn't a convenient way I can see without escape hatches for the code to access the permission from here

aws-cdk/packages/aws-cdk-lib/aws-secretsmanager/lib/rotation-schedule.ts

Line 103 in f22bd4e

props.rotationLambda.grantInvoke(new iam.ServicePrincipal('secretsmanager.amazonaws.com'));

[gavllew]

The grantInvoke returns a Grant instance, which implements IDependable, and offers an applyBefore method.

[peterwoodworth]

Ah neat, I never knew about this feature

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.