Search during index create

[henningandersen]

When creating an index, the primary will be either unassigned or
initializing for a short while. This causes issues when concurrent
processes does searches hitting those indices, either explicitly,
through patterns, aliases or data streams. This commit changes the
behavior to disregard shards where the primary is inactive due to
having just been created.

Closes #65846

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[henningandersen]

There is an edge case that needs handling before this can be reviewed. Sorry for the noise.

[elasticsearchmachine]

Hi @henningandersen, I've created a changelog YAML for you.

[javanna]

hi @henningandersen should we close this?

[droberts195]

should we close this?

Certainly the issue it was trying to fix, #65846, is still a problem that needs resolving somehow.

There is an edge case that needs handling before this can be reviewed

If that edge case is so difficult to fix that this whole change has to be scrapped then it would at least be good to write what it is on the issue so that any future attempts can try to avoid it.

[droberts195]

There is an edge case that needs handling before this can be reviewed.

What was the edge case?

The non-test code changes in this PR are very small, so it's a shame if this fix is completely blocked.

I am wondering if internal clients could be allowed to opt to ignore the edge case if it is not too bad, or if we're careful where we opt to ignore it. That could be done using a request parameter that's not exposed on the REST layer so that it's limited to internal use cases.

[henningandersen]

Thanks for bringing this up again David. The edge case is something like this:

1. A user creates an index without waiting for it to be green/yellow (or there could be an edge case with waiting too).
2. A user indexes data and refreshes it against the shard, before all nodes knows it is started.
3. A user searches the index, unluckily hitting a node that does not know the shard is started yet. The search returns without the data indexed in step 2.

It will be a rare edge case. I need to double check some things to see if it can also happen if we wait for yellow.

[maxhniebergall]

Hi team, is there any chance that this PR can get tested and merged?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)