[Lens] Field existence endpoint uses three APIs instead of one

[wylieconlon]

This is a tech debt issue with some overlap with Discover and index patterns in general. Both Lens and Discover show a list of fields on the left hand side, but the mechanism is different.

For Lens, the field list and existence information requires three steps (steps 1-3) that could be combined into one step:

1. Request the index pattern saved object
2. Request the index mappings
3. Request the field capabilities
4. Load 500 documents from the index
5. Determine if the field exists by recursively looking through _source

Steps 1-3 are necessary because there is no unified API that can return:

• Scripted fields which are persisted in the index pattern
• Alias information which is only available in the mappings
• The path in the _source document to the field data, especially for keyword fields

This is partly addressed by elastic/elasticsearch#49264 but not fully.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[timroes]

cc @mattkime @ppisljar This sounds to me like we would want to extend the index pattern API to retrieve all those information in it. alias support is tracked in #28570 and I would assume that once we have that source_path information in the ES API, we could also leverage that information in the index pattern API as well.

@wylieconlon Could you maybe clarify a bit, what part we are looking for that's not addressed by the linked ES issue, since we can still provide feedback on that API, and what we would need.

[wylieconlon]

@timroes Yes, the linked ES API would easily become part of the index pattern object and should solve both alias and multi use cases for us. I don't think any current index patterns have that information.

[wylieconlon]

The fewest requests we can do is 2:

1. Load the index pattern
2. Make a request to the _search API, if we can get it to return fields using dot paths: A high level way of retrieving values for certain fields elasticsearch#49028

[wylieconlon]

One of the performance optimizations we should be making is implemented in elastic/elasticsearch#57276 - because we are using time filtering this could be a big improvement for many queries.