(custom-resources): onDelete is called before onCreate is done

[magidnadav]

Describe the bug

The custom resource lambda gets called with a delete event before the create event lambda ended.

When a stack containing a custom resource failed to deploy for reasons that are not related to the custom resource.
cloud formation canceled the creation of the custom resource ("Resource creation cancelled") and then called the lambda with a delete event.
This called can happen while a different instance of the lambda performs the create flow which can cause an unstable state.

Expected Behavior

CDK will wait until the create flow is done before starting the delete flow

Current Behavior

CDK logs:

[2023-06-26T19:14:29.759Z] | 7:13:34 PM | CREATE_IN_PROGRESS | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[2023-06-26T19:14:29.762Z] | 7:13:38 PM | CREATE_FAILED | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51) Resource creation cancelled

[2023-06-26T19:14:29.762Z] | 7:13:40 PM | DELETE_IN_PROGRESS | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

[2023-06-26T19:14:29.769Z] | 7:14:08 PM | DELETE_COMPLETE | AWS::CloudFormation::CustomResource | (fixrds358createrdstableintegrationsCreateTableAndUser76462F51)

cloud watch logs:

The create lambda invocation:
2023-06-26T19:13:38.438Z 7f69edde-d3f6-4dd1-ae40-086b4d765112 INFO { RequestType: 'Create', ...}
...
2023-06-26T12:14:07.998-07:00 | REPORT RequestId: 7f69edde-d3f6-4dd1-ae40-086b4d765112

The delete lambda invocation:
2023-06-26T12:13:44.680-07:00 | 2023-06-26T19:13:44.680Z 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba INFO { RequestType: 'Delete', ...}
...
23-06-26T12:14:06.259-07:00 | REPORT RequestId: 406167ae-a7c6-4ce1-80bd-6d9b9dc2f2ba

Notice that the delete lambda start before the create lambda starts.

Reproduction Steps

I will try to create one soon

Possible Solution

Do not initiate the custom resource delete flow until it was created.

Additional Information/Context

No response

CDK CLI Version

2.75.0

Framework Version

No response

Node.js Version

18

OS

Ubuntu

Language

Typescript

Language Version

No response

Other information

No response

[peterwoodworth]

This sounds like a bug with CloudFormation, I recommend reaching out on the CloudFormation Coverage Roadmap.