[Discover] Switch from _source to fields when fetching data

[kertal]

Currently data displayed in Discovers grid is taken from _source (except scripted fields and date & date_nanos fields, which are requested by using docvalue_fields). For this reason multi fields (e.g. field.keyword), alias field, and the upcoming new kid on the field block called runtime field #77107 can be selected, but there's no value displayed in the document table.

How much adaptions are necessary depends on how data is returned when using fields, currently fields is returned destructured, but since Discover is currently using _source which is structured this would be a very different behavior.

What parts of Discover would need adaptation, when fields is ready to usage?

• Data fetching at the main document table [Discover] Use fields API to retrieve fields #83891
• Data fetching at the context document table
• Data fetching at the doc view (There's no requirement to [Data Services] Support fields parameter in high level search apis #77241 here)
• Displaying multi fields in the doc view [Discover] Migrate doc view to use ES search fields  #85112
• Initially Discover is displaying _source, it needs to be decided and implemented what should be displayed instead

Fields API change checklist

• Support unmapped fields
• Multi fields in extended document view
• Support nested fields unflattened (waiting for ES API)
• Make individual document viewer work with fields API
• Improve "visualize" button for multi-fields
• Handling of meta fields in the sidebar

Requirements:

Support request for fields in high level search service. #77241

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[lukeelmers]

Now that #82383 has merged, this should be unblocked 🙂

[timroes]

As of the latest discussion, ES team (@cbuescher) will begin work on the fields API to represent nested fields in query in an unflattened way. As soon as we have something ready, let's link it here. I've outlined the major expectations to the API from our side:

• Nested fields should be returned as structured (JSON) content from the closest to root nested object.
• Ideally (read optionally) they should not ALSO appear flattened, otherwise we'd need to filter out the flattened version in Discover.
• Whether this is default behavior or requires some request parameter is up to ES to decide.
• This behavior is specific for nested fields and no other type.

[timroes]

Closed via #83891 and more, tracking remaining work in #88426