cli: --no-rollback and a changeset that includes replacement type updates causes a failed deployment

[tmokmss]

Describe the feature

CDK should automatically disable no-rollback flag when a change set to deploy contains a replacement type updates. Otherwise the deployment fails and it leaves the stack in a non-terminal state, which does not allow updating stack any more without resetting the state.

Use Case

I'm always frustrated when I accidentally deploy a change with no-rollback flag enabled, and the deployment fails after a while because it contains replacement type updates.

Proposed Solution

CDK CLI uses CFn changeset feature to deploy a change, so it knows whether the change contains any replacement. If there is, it should automatically disable no-rollback flag.

Or we should at least validate the flag before actually executing a CFn deployment.

Other Information

The error we get when deploying a change with no-rollback flag and replacement change:

Replacement type updates not supported on stack with disable-rollback.

The error we get when deploying a change after a deployment failed due to the above error.

This stack is currently in a non-terminal [UPDATE_FAILED] state. To update the stack from this state, please use the disable-rollback parameter with update-stack API. To rollback to the last known good state, use the rollback-stack API

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.145.0

Environment details (OS name and version, etc.)

macOS

[khushail]

Thanks for reaching out with this request @tmokmss . Marking it appropriate for team's input.

[rix0rrr]

To confirm: you're requesting that as a user, you pass in --no-rollback, but the tool silently ignores that flag because the change set contains a replacement?

[tmokmss]

Hi @rix0rrr, thank you for taking care of the FR. I think there are several options to improve the current behavior when there is a replacement type change and --no-rollback flag is set:

1. silently ignore --no-rollback, and deploy it with rollback
2. throw an error before actually deploying the changeset
3. CDK CLI automatically recovers from UPDATE_FAILED state

the 3rd option does not directly resolves the issue, but it reduces the manual work to recover from the following state:

This stack is currently in a non-terminal [UPDATE_FAILED] state. To update the stack from this state, please use the disable-rollback parameter with update-stack API. To rollback to the last known good state, use the rollback-stack API

[rix0rrr]

I think I would prefer doing something with handling the UPDATE_FAILED state. To that end, I'm going to do the following:

• Add a cdk rollback command to be able to get to a stable position from one of the paused fail states.
• For cdk deploy:
  • If the stack is currently in a paused fail state AND (--no-rollback is not specified OR a replacement is detected): prompt the user to roll back first before starting the deployment
  • If the stack is not currently in a paused fail state AND --no-rollback is specified AND a replacement is detected: notify the user that --no-rollback won't work when there are replacements to do, and prompt them to continue with a regular deployment.

None of this replacement detection will work with --deployment-method=direct, only with change sets.

[tmokmss]

@rix0rrr That makes sense, thanks! I think the last one is the highest priority, because we should not get into UPDATE_FAILED state very often if CDK CLI can successfully prevent users from deploying a stack with --no-rollback and replacement changes. (I assume the number of people using --deployment-method=direct is small).