Add a configurable delay for non-assigned shards in search request

[jimczi]

Today when no shards are available for a specific index in a search request, the shard is marked as failed and the failure is added in the search response.
This can be problematic for shards that have no replica since any crash/restart of a node can make several search requests to fail since the re-allocation of the shards will take some time. In the context of searchable snapshots, the re-allocation shouldn't take long (since the data is not entirely copied) so the initial idea is to rely on fast re-allocation rather than replicas. For this to work seamlessly for users, we thought that it could be helpful to add a configurable delay for search requests that would define the maximum amount of time for the request to wait for a shard to become active.
The idea would be to add a listener in the search request that would be notified when the shard is assigned to a new node and ready for requests. If the timeout is reached, a failure would be reported exactly the same as what we do today.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Recovery)

[jimczi]

We discussed this issue with the @elastic/es-distributed team and agreed that this change is a must-have for searchable snapshots availability. We also agreed that we can implement a generic solution for unassigned shards, not only searchable snapshots, which simplifies the integration in the search layer.
@ywelsch brought the idea of limiting the delay to unassigned primary shards, which eliminates the issues with yellow cluster that have unassigned replicas.
This solution seems promising so we'll try to implement it in the search action.

We also talked about shard searches that got interrupted because the node is stopped/restarted and agreed that we should also raise the priority of #56045. These two issues are linked since the delay could be applicable to retries in case an assigned shards becomes unassigned during execution.

[fcofdez]

@tlrx and I have discussed this issue and it seems like we won't pursue with this in the foreseeable future. It might be possible that something like the enhancement described in this issue could be useful in future features, but if that time comes, we think it would be better to open a new issue tailored to these needs.