Create index pattern page is slow if cluster is busy

[jakommo]

Kibana version: 6.4
Elasticsearch version: 6.4

When creating a new index pattern and the Elasticsearch cluster is busy and/or overloaded with too many shards, the query that is used to populate the list of indices can take a long time.
Currently the query looks like:

GET */_search
{"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}

On a cluster with several thousands of shards this can become expensive and slow.

I was wondering why we aren't using something cheaper like _cat/indices, but @pickypg already enlightened me that this would require cluster monitor privileges, which a normal Kibana user might not have. Running the agg also ensures that only indices are listed that the user has read access to, which might not be the case on the _cat API call.

From my discussion with @pickypg , we should at least think about setting shard_size on this request.
But I'd like to start a general discussion if there are better ways to get that list of indices the user has read access to.

[tylersmalley]

@elastic/kibana-management

[pickypg]

This is also why we don't see aliases (#24088).

[AndrewMcQuerry]

We ran into this issue recently, but it wasn't due to cluster being too busy. We had actually set the action.search.shard_count.limit to a lower value than the total number of shards in the cluster, so as to prevent folks from using wildcard Index Patterns in either Watcher or TSVB. However, when we made that decision, we didn't realize that this would cause the Index Pattern Create page to fail to query which indices existed in the cluster.

For Standard Index Patterns, we were bypassing this by selecting the "Include System Indices", which effectively doesn't try to perform the search until you type at least one letter.

But, for Rollup Index Patterns, this work-around doesn't work, since there is no "Include System Indices" on the "Create rollup index pattern" page. The page simply fails with "Couldn't find any Elasticsearch data" and "Failed to load indices".

It's not ideal that we are forced to increase action.search.shard_count.limit just to create a Rollup Index Pattern...

[spalger]

I think compatibility with cross-cluster search was also a part of this decision.

[mattkime]

I'm closing this as we use the /_resolve endpoint as of v7.9 #70271 - performance should be significantly improved for this case.