[Reporting CSV] Show Runtime Fields in CSV Export

[tsullivan]

Goal: CSV Export will have the same state of support for runtime fields that Discover supports, in every release.

The Discover effort is a phased approach which starts in 7.11 and planned to last to 7.12. To meet the stated goal, this issue must have the same priority or higher as the Discover App effort.

Terminology

Refer to "Show runtime field values in Discover" (#77107) for context and terminology. CSV Export export of Discover's Data Grid view in a text-only format. For Kibana Reporting, "export" means running a job in the server background, archiving the results in ES, and making file download available any time to the user.

Intro

Recap of the Discover effort on this issue. Runtime fields in Discover affect:

• Unmapped fields do not appear in the fields info that includes runtime fields.
  • Discover will have to use data from hit.fields as well as hit._source to show all the data that the user expects.
• Object arrays will appear differently
• Multi-fields appear differently.

Proposal

Rewrite CSV Export to use new Data plugin APIs. Assuming that Discover is using the same APIs, this makes the logic shared between CSV Export and Discover. That has been a long-time goal which can finally come to fruition with the SearchSource API available on the server.

Level of Effort

The goal of the API is to return a flattened and formatted data grid when given:

• saved search ID
• space ID
• global time picker filter value
• client time zone

Reporting CSV Export will call this API repeatedly to scroll the data from the index, up until the maxSizeBytes limit is hit.

Backwards Compatibility

Reporting has an interesting angle to BWC, because users may be sending requests to Kibana using a POST URL that was generated in any unknown version. For CSV Export, the POST URLs include JSON from the index pattern saved object, as well as time-relevant fields like the time range from the timepicker and client time zone.

[tsullivan]

POC to use SearchSource for CSV Export

#81446

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)