[Transform] use ISO dates in output instead of epoch millis

[hendrikmuhs]

Transform writes dates as epoch millis, this does not work for historic data in some cases or is unsupported. Dates should be written as such. With this PR transform starts writing dates in ISO format, but as existing transform might rely on the format it provides backwards compatibility for old jobs as well as a setting to write dates as epoch millis.

fixes #63787

Example

The following example illustrates the change, we use the following config for pivot (dataset: kibana_sample_data_logs):

  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1m"
        }
      }
    },
    "aggregations": {
      "latest_timestamp": {
        "max": {
          "field": "@timestamp"
        }
      },
      "agent_dc": {
        "cardinality": {
          "field": "agent.keyword"
        }
      }
    }

The change affects how @timestamp is written in _source (note: dates in the aggs part are already written as ISO string, with the change dates in pivot.group_by and pivot.aggregations behave consistently):

Before

        "_index" : "t3",
        "_id" : "ANAnyse5SVpHo-UXL4f8598AAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : 1606005540000,
          "latest_timestamp" : "2020-11-22T00:39:02.912Z",
          "agent_dc" : 1
        }

After

        "_index" : "t4",
        "_id" : "ANAnyse5SVpHo-UXL4f8598AAAAAAAAA",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-11-22T00:39:00.000Z",
          "latest_timestamp" : "2020-11-22T00:39:02.912Z",
          "agent_dc" : 1
        }

Doc values do not change, because either way it is parsed as date (but parsing can't fail anymore if dates <1970).

In case you rely on the old format (because you use the produced _source in an application), you can get back the old style with:

  "settings": {
    "dates_as_epoch_millis": true
  }

When updating an old transform using _update, this is automatically set. If you want to migrate an old transform to the new style, you can use _update to explicitly set dates_as_epoch_millis to false or null (== default).

Docs preview for the added parameter: https://elasticsearch_65584.docs-preview.app.elstc.co/diff

Post PR actions:

• document change in "Breaking changes" for 7.11
• BWC

[hendrikmuhs]

@elasticmachine update branch

[hendrikmuhs]

@szabosteve Can you have a look at the doc changes: elasticsearch_65584.docs-preview.app.elstc.co/diff

[szabosteve]

Docs changes LGTM, thanks.
It's just a nit, so please take it or leave it: we try to order the parameters alphabetically in the API docs and in the common params file (I know, we are not always consistent...). If you don't have time for changing it, no problem, common-parms needs to be reviewed anyway soon.

[hendrikmuhs]

It's just a nit, so please take it or leave it: we try to order the parameters alphabetically

I fixed it. I blame the last minute rename of the parameter, the 1st version was write_da....

[przemekwitek]

LGTM

with just two small remarks

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)