rds: UPDATE_ROLLBACK_FAILED when deploy while "storage change is being optimized"

[anentropic]

Describe the bug

I tried to deploy a change to the allocated storage of my RDS, but it turns out the db was already in the process of autoscaling the storage

This resulted in the following error:

11:19:27 | UPDATE_FAILED        | AWS::RDS::DBInstance                        | DatabaseMySQLADA28B0A
Resource handler returned message: "You can't currently modify the storage of this DB instance because the previous storage change is being
optimized. (Service: Rds, Status Code: 400, Request ID: 47f80e3c-545e-4094-b31f-47267d87956e)" (RequestToken: d0baaf92-da6d-b0f9-258a-131f90
3fc67f, HandlerErrorCode: InvalidRequest)

and

UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [DatabaseMySQLRotationSingleUser27AA2177, DatabaseMySQLADA28B0A]. )
    at FullCloudFormationDeployment.monitorDeployment (/Users/anentropic/.nvm/versions/node/v18.18.0/lib/node_modules/aws-cdk/lib/index.js:430:10615)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/Users/anentropic/.nvm/versions/node/v18.18.0/lib/node_modules/aws-cdk/lib/index.js:433:200503)
    at async /Users/anentropic/.nvm/versions/node/v18.18.0/lib/node_modules/aws-cdk/lib/index.js:433:18134

Expected Behavior

It's ok for the deploy to fail but it is unacceptable to end up in UPDATE_ROLLBACK_FAILED state

if the deploy has to fail it should be possible to retry the deployment after the db has finished its storage task

(I tried deploy again later but I just get Error [ValidationError]: Stack:arn:aws:cloudformation:eu-west-1:570110252051:stack/ifm-ssa-reimport-dev-eu/a5bd21a0-ff31-11ee-854d-0a9cdce91997 is in UPDATE_ROLLBACK_FAILED state and can not be updated.)

Current Behavior

stack is in UPDATE_ROLLBACK_FAILED state

AFAICT there is no way to recover from this state so my only option is to destroy the stack and start over

that is not possible on a production deployment

Reproduction Steps

deploy a stack with an RDS db, with a max_allocated_storage configured to enable autoscaling

add data to the db until it runs out of space

change the stack definition to increase allocated_storage then attempt the deployment while the db is busy autoscaling

Possible Solution

there are so many ways to end up in UPDATE_ROLLBACK_FAILED state with CDK that I feel there must be something fundamentally wrong with the underlying design

I can't recommend this to anybody

Additional Information/Context

No response

CDK CLI Version

2.138.0 (build 6b41c8b)

Framework Version

No response

Node.js Version

v18.18.0

OS

macOS 14.4.1

Language

Python

Language Version

3.11.5

Other information

No response

[pahud]

I guess this is a CFN bug. I am cutting an internal ticket for clarifying.

[pahud]

internal tracking V1358326979

[pahud]

Hi

It's not a bug of CFN. It's how CFN works when the deploy fails and the update handler will just try to fall back to previous state.

Please use continue-update-rollback to get rid of the stuck.

[pahud]

closing as it's not a CDK issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.