r/resource_aws_subnet: increase deletion timeout

[s-urbaniak]

This adds a bump in the timeout for removing subnets as we experienced a high flake count upon cluster destruction. Introducing this timeout reduces flakes significantly

This is a stop-gap solution until resource providers get timeout override support.

Fixes coreos/tectonic-installer#1245

/cc @radeksimko @jasminSPC

[radeksimko]

See #1052 (comment)

[radeksimko]

Hi @s-urbaniak
as discussed in other places and conversations - raising the timeout should always be the last option unless the API explicitly exposes state (e.g. pending), so we know for sure it's worth waiting.

This isn't subnet's case, so I believe it's avoidable. We want to prevent users with genuine dependency violations from waiting too long until they see the error.

By looking at the resources you use in tectonic-installer I think there's a chance it was already addressed in either #1427 or #1036

Generally speaking one of the most common reasons I bumped into this error is when AWS service managing its own ENIs left some interfaces around after deleting the resource (e.g. VPC-enabled Lambda, ELB, ALB, ...). Unfortunately this is also the reason why it's so hard to find the root cause.

With that I'm going to close this issue, but I'd advise you to run the destruction part of your smoke tests like this:

aws ec2 describe-network-interfaces --filters=Name=vpc-id,Values=$VPC_ID | tee network-interfaces-before.json
terraform destroy ...
aws ec2 describe-network-interfaces --filters=Name=vpc-id,Values=$VPC_ID | tee network-interfaces-after.json

so that we can inspect ENIs present before and after deletion if you do bump into this error again.

[s-urbaniak]

@radeksimko thanks a lot for the heads-up! I am very glad and thankful to see this going forward, hence I am fine with closing this then. We are already using stock terraform without custom patches like this one in our pipeline now too.

/cc @cpanato @mxinden can we add Radek's hint to use aws ec2 describe-network-interfaces ... to our CI pipeline and add trace points in destroy failure loops to enable ENI inspection?

[mxinden]

@s-urbaniak coreos/tectonic-installer#2157 addresses this.

@radeksimko Thanks a lot for your help and advice!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!