add cluster.wait_for_workers (called by client.wait_for_workers if the client.client exists)

[dchudz]

Currently users call client.wait to wait for a certain number of workers.

Currently a common pattern is:

cluster.scale(100)
client.wait_for_workers(100)

Cluster managers (like Coiled) would like to include custom handling to give users nice information about what's happening (including especially failures) while waiting for workers. At the moment, this isn't possible.

Proposal to make this possible:

• add cluster.wait_for_workers (with logic similar to the existing client.wait_for_workers)
• client.wait_for_workers should call cluster.wait_for_workers if there's a cluster object.

Then cluster managers can override cluster.wait_for_workers to provide custom handling, which will be called whether the user calls cluster.wait_for_workers or client.wait_for_workers.

This suggestion arose from discussion with @jacobtomlinson in #6041.

[consideRatio]

• client.wait_for_workers should call cluster.wait_for_workers if there's a cluster object.

This should be if there is a cluster object that implements such function only right? I think this led to a failure in dask_gateway that hasn't yet added support for this - which it can, but it is currently broken unexpectedly until it does i think.

This was reported in dask/dask-gateway#782.

---

Should dask/distributed be updated to handle a cluster object not implementing this new feature? (besides updating dask-gateway to support it)