aws-cloudformation: CfnStackSet requires bootstrap in all regions and cannot find assets on different regions

[pedromtcosta]

Describe the bug

I have a CDK app which consists of:

• MainStack;
• ReplicaStack;
• ReplicaStackSet;

ReplicaStack and ReplicaStackSet are defined on the scope of MainStack. MainStack is deployed on eu-central-1 and CDK is bootstrapped on eu-central-1. When setting stackInstancesGroup.regions of ReplicaStackSet to ['eu-central-1', 'us-east-1'] I get the error:

[Unable to fetch parameters [/cdk-bootstrap/hnb659fds/version] from parameter store for this account.]

After bootstrapping us-east-1 as well and trying to deploy the app again, I get the error:

Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist

I believe this means that when deploying the instances of ReplicaStack, CloudFormation is looking for the assets on the region where the instance is being deployed, instead of the region where ReplicaStackSet is deployed.

Expected Behavior

The deploy should work and should create my MainStack, ReplicaStackSet on region eu-central-1 and two instances of ReplicaStack on the regions eu-central-1 and us-east-1

Current Behavior

Deploy fails with message [Unable to fetch parameters [/cdk-bootstrap/hnb659fds/version] from parameter store for this account.] when us-east-1 is not bootstrapped and Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist and it is bootstrapped

Reproduction Steps

Create ReplicaStack:

export class ReplicaStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props);

    new lambda_nodejs.NodejsFunction(this, "MyFunction", {
      entry: "src/my-function.ts",
      handler: "handler",
      runtime: lambda.Runtime.NODEJS_18_X,
      timeout: cdk.Duration.seconds(60),
      memorySize: 1024,
    });
  }
}

Create MainStack:

export class MainStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props);
}

define the app:

const app = new cdk.App();
const stack = new MainStack(app, 'MainStack');
const stage = new cdk.Stage(stack, 'Stage');

const replicaStack = new ReplicaStack(stage, 'ReplicaStack');

const stackTemplate = JSON.stringify(
  stage.synth().getStackByName(replicaStack.stackName).template
);

new cloudformation.CfnStackSet(stack, 'ReplicaStackSet', {
  permissionModel: 'SELF_MANAGED',
  stackSetName: 'ReplicaStackSet',
  templateBody: stackTemplate,
  capabilities: [
    'CAPABILITY_IAM',
  ],
  stackInstancesGroup: [
    {
      creationStack: [],
      deploymentTargets: {
        accounts: [config.env.aws.AWS_ACCOUNT()],
      },
      regions: ['eu-central-1', 'us-east-1'],
    }
  ]
});

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0 (build 9fca790)

Framework Version

No response

Node.js Version

v16.19.0

OS

MacOS Ventura 13.0

Language

Typescript

Language Version

No response

Other information

No response

[peterwoodworth]

This wouldn't work because the assets have to actually be uploaded - the code you have won't trigger assets to upload in any region.

Try out https://github.com/cdklabs/cdk-stacksets/

[pedromtcosta]

@peterwoodworth thank you for your reply, I tried this lib already but also got stuck as well, that's why I am trying out with the L1 constructs: cdklabs/cdk-stacksets#171

From my understanding, NodejsFunction should create the assets on the bucket, I have several projects using CDK and I never had to manually upload the Lambda assets

[peterwoodworth]

Assets are uploaded as part of each individual stack deployed through CDK. If a stack is being deployed through a CfnStackSet, CDK can't know that it should upload the assets to the regions defined in the CfnStackSet. I'm not sure if the hashes vary by region, if they do then that's another thing that needs to be accounted for when creating a stack set.

In short, if your stack has assets and you depend on CDK uploading those assets and defining their locations, CfnStackSet will never work. A higher level construct needs to handle that - it appears from the issue you linked as though the current solution has a bug or needs clarification in how to use it.

[pahud]

@pedromtcosta

Looks like you are trying to deploy the same stack to multiple regions. Why not just deploy like this:

const envUE1 = { region: 'us-east-1', account: process.env.CDK_DEFAULT_ACCOUNT };
const envEC1 = { region: 'eu-central-1', account: process.env.CDK_DEFAULT_ACCOUNT };

new ReplicaStack(stage, 'ReplicaStackUE1', { env: envUE1);
new ReplicaStack(stage, 'ReplicaStackEC1', { env: envEC1);

On cdk deploy --all, CDK will bundle your lambda assets and upload to 2 assets buckets in the two target regions and deploy them one by one.

[peterwoodworth]

Yes, hopefully this will suffice in the meantime. If you want to follow up with the issue with the current solution, the best place to discuss that will be in the repo for that construct where you already have an issue open.

If you desire to see this working as part of the official CDK library, that is being tracked as a feature request here.

If you need any more help getting something setup with the tools available, please create a discussion

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.