No longer accessing another stack's resource causes deadly embrace

[mcalello]

CDK's automatic determination and synthesis of exports to imports can get locked into a deadly embrace that cannot be resolved without destroying your stacks.

Reproduction Steps

Stack1 creates a common Security Group, sg1.
Stack2 uses sg1.

CDK's automatic determination and synthesis described here, will generate an export output from Stack1 for sg1 and an Fn:ImportValue into Stack2.

Then you deploy these stacks.

Later, you decide that Stack2 really needs it's own more specific Security Group, so you create it's own sg2. Again, CDK's automatic determination and synthesis realizes that sg1 is no longer referenced by Stack2 (or any other stack) and attempts to delete the Export of sg1 from Stack1.

This will fail and cannot be deployed.

CloudFormation will be prevented from deleting sg1 as an export from Stack1 since it is currently being used as an Import in the (existing) Stack2. Ironically, you were trying to update Stack2 to no longer reference Stack1's sg1.

Error Log

UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack  | Stack1 Export Stack1:ExportsOutputFnGetAtt-sg1-C785ABD5GroupId30372C66 cannot be deleted as it is in use by Stack2

Environment

• **CLI Version : aws-cli/1.16.223 Python/3.7.4 Darwin/18.7.0 botocore/1.12.213
• **Framework Version: 1.34.1 (build 7b21aa0)
• **OS : macOS 10.14.6
• **Language : Python3

Other

---

This is 🐛 Bug Report

[austinbv]

I have a bit more code to support this as you create a circular reference that ends up meaning NO changes can be made without deleting an entire stack

export class StackA extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    new Instance(this, "instance1", {
      allowAllOutbound: true,
      machineImage: MachineImage.genericLinux({
        "us-east-2": "ami-*****",
      }),
      blockDevices: [
        { deviceName: "/dev/sda1", volume: mainVolume },
        { deviceName: "/dev/sdm", volume: largeVolume },
      ],
      vpc,
      instanceType: InstanceType.of(InstanceClass.M5A, InstanceSize.XLARGE2),
    });

    new IamStack(this, "IAM stack", {
      instances: [instance],
      env: props?.env,
    });
  }
}

interface IamStackProps extends StackProps {
  instances: IInstance[];
}

export class IamStack extends Stack {
  constructor(scope: Construct, id: string, props: IamStackProps) {
    // ...
    this.group.addToPolicy(
      new PolicyStatement({
        actions: ["ec2-instance-connect:SendSSHPublicKey"],
        effect: Effect.ALLOW,
        resources: props.instances.map(this.buildArn),
      })
    );
    // ...
  }
  private buildArn(instance: IInstance): string {
    return `arn:${Aws.PARTITION}:ec2:${Aws.REGION}:${Aws.ACCOUNT_ID}:instance/${instance.instanceId}`;
  }
}

No change can be made to the instance resource that would require it to recreate (update the AMI). And no change can be made to the Policy statement (including deleting it).

The error is StackA Export StackA:ExportsOutputRefinstance4096131FFAFEED01 cannot be deleted as it is in use by IAMstackF7BED611

The only want to solve this problem is to delete the dependent stack. Deploy the parent stack. Add back the dependent stack.

[austinbv]

so I found a way around this.

If you are updating the child stack you can deploy with the -e tag not deploy the parent stack.

If you are changing the parent stack it's a 2 step process.

1. Remove the dependency on the parent stack. In my case it was remove the PolicyStatement

export class IamStack extends Stack {
  constructor(scope: Construct, id: string, props: IamStackProps) {
    // ...
   // this.group.addToPolicy(
    //  new PolicyStatement({
    //    actions: ["ec2-instance-connect:SendSSHPublicKey"],
    //    effect: Effect.ALLOW,
    //    resources: props.instances.map(this.buildArn),
    //  })
    //);
    // ...
  }
  private buildArn(instance: IInstance): string {
    return `arn:${Aws.PARTITION}:ec2:${Aws.REGION}:${Aws.ACCOUNT_ID}:instance/${instance.instanceId}`;
  }
}

Then deploy the IAMStack with the -e tag.

2. Change the parent ami and uncomment the IAMStack and deploy the whole stack.

[eladb]

We are aware of this issue (duplicate #3414), but I am not sure exactly what we can do about it. Any ideas?

[austinbv]

Our solution is to deploy the substack with the -e flag then deploy the parent stack.

Even some documentation around that would be stronger than just two immutable stacks. Could you always update leaf nodes first, then update the full stack again

[mcalello]

@eladb We hit this problem again today. After thinking through it again, we are wondering... could you mark the no longer referenced output for deletion, but not actually delete it in this pass. Then in a subsequent deploy, it could be safely deleted?

[eladb]

The CDK (currently) doesn't have knowledge of the current state, only the desired state. The exports/imports that are created as a result of cross stack references are only defined if the reference exists. Otherwise they are simply not defined in your CDK app and we don't know that they existed there before.

[smeyffret]

Could we force the export somehow, even without the resource? If I understand correctly the export name is only based on the path of the construct and the type of reference we're using, not the actual construct itself (please correct me if I'm wrong). Could we have an helper method that will generate the export name for us, so that we can manually keep the export while removing the resource?

const { exportId, exportName } = helperMethodToGenerateImplicitExportName({
    scope: this,
    constructIdToExport: legacyConstructId,
    typeOfReference: 'Ref', // could be 'Arn' or any other type of references
});
new CfnOutput(this, exportId, {
    exportName: exportName,
    value: 'dummy_value',
    description: 'Dummy export to prevent deadly embrace',
});

The method would generate the name StackA:ExportsOutputRefinstance4096131FFAFEED01 for us, thus allowing to remove the resource.

If we need to keep the value untouched (instead of hardcoding 'dummy_value' here), could we use AwsCustomResource to query the existing value of this export and return it as part of the helper method as well?

I personally have to add those exports manually when I am in this situation (hardcoded in the code), and I have a helper script to help query the names from CloudFormation for all stacks. But a dynamic way of generating them would go a long way.

[Dzhuneyt]

We encountered this unsolvable case in a trivial infrastructure:

RDS stack -> (referenced in) -> ECS stack

This causes CDK to create CloudFormation outputs and inputs between the two stacks, which is fine. However, at some point, we decided to drop this dependency in favor of another approach of retrieving Database credentials (using Secrets Manager, etc).

This caused CDK to fail though because it first detects that the RDS stack has no longer any dependencies and tries to delete the exported output, but at the same time - fails to do so - because the ECS stack is currently using that output (even though our desired effect is that it no longer uses it.

Is there an elegant solution to this? Maybe one where CDK is smart enough to first deploy the ECS stack to clean up the link between the two?

[skinny85]

@Dzhuneyt while dealing with removing cross-stack references can be tricky in the CDK, there is actually a way to do it. Check out this article: https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references for a step-by-step guide on how to unblock yourself.

Thanks,
Adam

[edisongustavo]

I have a workaround here too: #5819 (comment)

[eikeon]

@Dzhuneyt while dealing with removing cross-stack references can be tricky in the CDK, there is actually a way to do it. Check out this article: https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references for a step-by-step guide on how to unblock yourself.

Thanks,
Adam

@skinny85, Thank you for your post. Do you happen to have an example of creating the dummy resource in the case one’s stack is in a CDK pipeline?

[skinny85]

@Dzhuneyt while dealing with removing cross-stack references can be tricky in the CDK, there is actually a way to do it. Check out this article: https://www.endoflineblog.com/cdk-tips-03-how-to-unblock-cross-stack-references for a step-by-step guide on how to unblock yourself.
Thanks,
Adam

@skinny85, Thank you for your post. Do you happen to have an example of creating the dummy resource in the case one’s stack is in a CDK pipeline?

It's exactly the same. You just git push after creating the dummy Outputs instead of running cdk deploy.

[eikeon]

Coming up with all the values to create the dummies for the various stages the stack appears in is not exactly the same from my perspective. But think I've dug deep enough to create a function that'll compute them... what @smeyffret was suggesting. Seems like that would be a low hanging fruit option to offer up to help people navigate this deadly embrace while something more magical is worked out. @eladb @rix0rrr has the explicitly declare the output option been considered -- from above: #7602 (comment)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.