Show runtime field values in Discover

[stacey-gammon]

Terminology

For clarity, the following terms are used to describe the following items in a discover page (don't confuse table with grid):

Intro

Runtime fields that are defined on an index mapping should be usable in Discover. The current experience is that discover thinks the fields have no values so hides them by default, and values won't appear in the grid if they are added as columns. Searching will still work, so this creates a confusing UX experience.

This happens because discover accesses these values from _source, and _source doesn't include runtime fields. Discover should use the new search fields parameter to retrieve these values.

There are some differences when switching to use the fields array:

• The fields array won't contain unmapped fields (but they do show up in _source)
• Object arrays are flattened, where previously they weren't.
• Multi-fields will appear twice in the fields array (where in _source they will only show up once).

After reviewing our options, we'd like to propose the following ideal discover experience that avoids breaking changes and minimizes performance issues.

Ideal (non breaking) Discover UX

Field list

• Runtime fields appear in the field list in the left hand side (automatically - user doesn't have to explicitly add them).
• Multi fields appear as they do today (by default one field is displayed, but you can view them all using the "hide missing fields" option).

Query bar

• Runtime fields appear in KQL suggestions.
• Runtime fields can be used in filters and queries.

Data grid

• Users can add runtime fields to the data grid, which will show the correct value.
• Formatting of all fields remains the same (nested fields and object arrays are unflattened).
• ** Unmapped field values will no longer show up in the field list by default. We will need to support this as a per saved search setting to avoid a breaking change. It will be off by default for any saved searches that aren't already using unmapped field values as columns in the data grid. This would appear as a new option, "Show unmapped fields" in this pop up box:

Expanded doc view

• Users will see runtime fields and their values in the table (whether the field is a column in the grid or not).
• Users will see runtime fields and their values in the JSON view (whether the field is a column in the grid or not).
• Formatting of all fields remains the same (nested fields and object arrays are unflattened).
• Multi fields appear as they do today (only one field is ever displayed).
• Unmapped fields continue to appear in both table and JSON view (whether this is ideal or not is debatable, but it avoids a breaking change).

Implementation details

• Fields array would need to avoid flattening arrays of objects, either by default, or support a parameter to specify this.
• If "show unmapped field values" is off, Discover will only request field values for fields in the grid, and will explicitly turn off _source to avoid the performance penalty of parsing twice as large of a JSON object. If it is on, discover will ask for both, at the cost of a performance penalty. How bad is TBD.
• A separate request will be sent out to create the expanded doc views, but only on demand. This will slow down the loading of this table. This is an acceptable tradeoff.
• Discover will use information from the Index Pattern to collapse multi-fields manually (note: we should follow up with solution teams to see if this is going to be a pattern of usage, and if so, bake it into the fields array as well).
• For existing saved searches, discover will need to determine if any columns are unmapped fields and turn this "show unmapped fields" toggle on to avoid a breaking change. TODO: determine feasibility of this logic.

Blockers

• App arch services will need to support the ability to use and/or configure the fields parameter. In Discover, be able to show doc_values for any field not stored #64878

Open questions

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[timroes]

Closed by #83891 and others, tracking remaining work in #88426