want a way for background tasks to await saga completion

[hawkw]

Nexus API handlers (and other sagas) that start sagas have the ability to wait for the completion of the saga by awaiting the RunningSaga future returned by RunnableSaga::start. Background tasks, on the other hand, cannot currently do this, as their only interface to the saga executor is an Arc<dyn StartSaga>, which provides only the following saga_start method:

omicron/nexus/src/app/saga.rs

Lines 96 to 108 in 8be99b0

	impl StartSaga for SagaExecutor {
	fn saga_start(&self, dag: SagaDag) -> BoxFuture<'_, Result<(), Error>> {
	async move {
	let runnable_saga = self.saga_prepare(dag).await?;
	// start() returns a future that can be used to wait for the saga to
	// complete. We don't need that here. (Cancelling this has no
	// effect on the running saga.)
	let _ = runnable_saga.start().await?;
	Ok(())
	}
	.boxed()
	}
	}

Note that the RunningSaga future is thrown away, meaning that the saga has been started, but there's no way to wait for its completion.

It would be useful for background tasks to be able to wait for the saga's completion. Not being able to do means that background tasks don't really have the ability to enforce meaningful concurrency limits on the sagas they start --- they can only limit the concurrency of starting sagas, not the number of sagas that are concurrently running. It also means that the approach for avoiding Nexus replicas all operating on instances/VMMs/disks in the same order described by @gjcolombo in #6565 (comment) is much harder to implement in cases where the operation that changes the state of a database record so that it is no longer returned by a query requires a saga completing, since the background task cannot easily wait for the sagas it starts to finish before querying the database again.

See also 0582e84

[hawkw]

I'd like to extend the StartSaga interface to permit it to return a saga completion future. Ideally, this would just be the RunningSaga type, but unfortunately, a number of tests currently rely on a NoopStartSaga type that doesn't actually start real sagas, so we'd need to abstract over that somehow...