Provide an option to only assign shards to nodes that already have them as part of allocation

[ppf2]

Consider the following repro:

• node 2 has [0], [1], [4] as its original allocation.
• cluster.routing.allocation.node_concurrent_recoveries is set to 1 (on purpose for the reproduction).
• allocation is disabled, and then node 2 is stopped.
• node 2 is started back up and then allocation is enabled again.
• [0] and [1] get allocated back to node 2 successfully when allocation is enabled (after its restart)
• [4] ends up on node 1 (instead of node 2) because node 2 already has 1 target recovery outstanding (so it decides to find another node to allocate [4] to while node 2 is performing its recovery) while node 1 has 1 source recovery (but capable of being a target for the recovery of [4]).
• At the end, rebalancing kicks in and moved [2] from node 1 to node 2, so node 2 now has [0], [1], and [2] instead of [0], [1], [4] .

It will be helpful to provide an additional option to cluster.routing.allocation.enable so that it will only assign shards to nodes that already have them to prevent it from performing unnecessary allocation of a shard to a different node as part of rolling restarts. While increasing cluster.routing.allocation.node_concurrent_recoveries (from the default of 2) is a potential workaround for small deployments, it is not a viable solution for deployments with a large # of shards on each node due to its potential network and i/o implications. For example, we can add an existing option to cluster.routing.allocation.enable that also works in conjunction with settings like new_primaries (eg. "existing,new_primaries”).

[martijnvg]

@ppf2 I think delayed shard allocation already helps here? and #12421 would kind of do what is desired in an automatic manner?

[clintongormley]

I agree with @martijnvg. Adding hard rules will break allocation in ways that could lose data. Closing in favour of #12421 and #11438