Remove "expand index pattern when searching" setting for index patterns

[spalger]

Kibana needs to completely remove use of the _field_stats API as it is about to be removed from elasticsearch.

With elastic/elasticsearch#25658, Elasticsearch now has the built-in behavior of automatically optimizing search/msearch requests to only hit shards that could potentially have documents that match the given filters. This is almost the same behavior that Kibana was hacking together with a pre-flight request to field_stats, so no alternate solution in Kibana itself should be necessary any longer, and we can just outright remove the pre-flight behavior.

[epixa]

The Elasticsearch issue for reference: elastic/elasticsearch#25577

[epixa]

Some details about the original motivation for deprecating field_stats can be found here: elastic/elasticsearch#23914

[spalger]

I guess another option would be to do a search in place of _field_stats, including just the time filter and a terms agg for _index (maybe with a size of 1000 and failing if more indices are selected?)

[trevan]

@spalger, I don't see how _field_stats helps with your example (1 index with 1000 shards). Because _field_stats would just match that 1 index and you'd still do the 20,000 requests. Am I mistaken?

[spalger]

You're right @trevan, I meant an index pattern like logstash-*. Updated the description to hopefully be more clear

[epixa]

This elasticsearch PR may be the ticket to removing this opt-in pre-flight behavior entirely: elastic/elasticsearch#25658

[epixa]

We've nailed down a solution here, but I want to preserve the original ticket description in case we need to revisit this:

The reason we can't just rely on elasticsearch to optimize the queries right now is that we use _msearch, which effectively executes all requests in their own context and doesn't share shard resolution amongst the requests. Because of this, if an index pattern addresses 1000 shards and there are 20 visualizations for that index pattern on a dashboard, elasticsearch will end up doing 20,000 requests, one to each shard. Those requests should be pretty quick because of optimizations in Elasticsearch, but 20,000 requests for a single dashboard is still too many.

A couple of ideas have been floated:

• move away from _msearch and somehow merge all searches into a single search request, potentially with a top-level filters agg to segregate the results from multiple visualizations
  • this would require a pretty massive amount of changes to the way that Kibana queries, which is possible but too much work to get into 6.0
• add the ability to share some sort of context in msearch requests, potentially by the user supplying a query that is globally applied and would result in a shard subset that all of the _msearch requests would be executed against
  • probably a pretty massive change to the msearch API in elasticsearch which is also unlikely to go into 6.0