Fix context handling in WaitForVolumeAttachment & add in-flight checks to attachment/detachment operations

[torredil]

What is this PR about? / Why do we need it?

cloud.go changes:

This PR addresses incorrect handling of the Context in WaitForAttachmentState which implements a retry mechanism using exponential backoff to wait for an attachment to reach the expected state via the DescribeVolumesWithContext API.

The context is used to carry deadlines and cancellation signals across API boundaries. When polling for volume state, its possible for the context to be cancelled due to timeout etc. If this happens while we're waiting for the volume to reach the expected state, it is desirable to stop waiting and immediately return an error to the caller. This change allows us to better handle cases where the context is cancelled or reaches its deadline by stopping execution when no longer necessary.

---

controller.go changes:

Added in-flight checks to both ControllerPublishVolume and ControllerUnpublishVolume. The in-flight check acts as a synchronization mechanism by providing mutual exclusion to ensure idempotency.

---

Testing changes:

Added tests to validate code paths and logic when attaching / detaching volumes. Also did some refactoring to move towards table-driven tests which vastly improves readability and maintainability of these tests.

What testing is done?

• Manual
• CI

With this patch:

I0602 17:16:29.032300       1 controller.go:370] "ControllerPublishVolume: attached" volumeID="vol-0163b0b755696dcdc" nodeID="i-07532540ce852721b" devicePath="/dev/xvdal"
I0602 17:16:47.545514       1 controller.go:362] "ControllerPublishVolume: attaching" volumeID="vol-0bf9e9d3205eb9fe7" nodeID="i-07532540ce852721b"
I0602 17:16:49.352063       1 controller.go:370] "ControllerPublishVolume: attached" volumeID="vol-0bf9e9d3205eb9fe7" nodeID="i-07532540ce852721b" devicePath="/dev/xvdap"
I0602 17:16:49.358931       1 controller.go:362] "ControllerPublishVolume: attaching" volumeID="vol-0bf9e9d3205eb9fe7" nodeID="i-07532540ce852721b"
I0602 17:16:49.519855       1 controller.go:370] "ControllerPublishVolume: attached" volumeID="vol-0bf9e9d3205eb9fe7" nodeID="i-07532540ce852721b" devicePath="/dev/xvdap"

Without this patch:

I0524 20:56:35.690733       1 inflight.go:74] "Node Service: volume operation finished" key="snapshot-299d9d27-a925-4954-93df-80ccd2105e85"
I0524 20:56:35.711511       1 cloud.go:713] "Waiting for volume state" volumeID="vol-05ee7370b8091e868" actual="detaching" desired="detached"
I0524 20:56:36.174965       1 inflight.go:74] "Node Service: volume operation finished" key="pvc-2a9b6b41-dc69-42ff-aa57-ec6bfcbd73e0"
I0524 20:56:36.176699       1 cloud.go:713] "Waiting for volume state" volumeID="vol-08a1528c35dfeb1ff" actual="detaching" desired="detached"
I0524 20:56:36.205657       1 cloud.go:654] "Ignoring error from describe volume, will retry" volumeID="vol-0aecdf207857b8263" err=<
	RequestCanceled: request context canceled
	caused by: context deadline exceeded
 >
I0524 20:57:31.4134426       1 cloud.go:654] "Ignoring error from describe volume, will retry" volumeID="vol-0aecdf207857b8263" err=<
	RequestCanceled: request context canceled
	caused by: context deadline exceeded
 >
...

[torredil]

/retest

[ConnorJC3]

lgtm

[hanyuel]

/lgtm

[ConnorJC3]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ConnorJC3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment