Automatically migrate default ILM policy to Data Stream Lifecycle Management

[axw]

In 8.15.0 we enable the apm-data plugin by default in Elasticsearch, which ships index templates that make use of data stream lifecycle by default. It is still possible to opt into ILM (except in Serverless projects, which do not support ILM).

There was a bug in 8.15.0 that we fixed in 8.15.1 (#13898): data streams upgraded from older versions would be "unmanaged" on upgrade. Data stream lifecycle is only automatically configured on newly data streams, so we need to keep some ILM configuration for existing data streams.

This leaves us in an indefinite undesirable state:

• new data streams use data stream lifecycle
• old data streams use ILM, unless manually migrated

We should automatically migrate existing data streams from ILM to data stream lifecycle management when the existing data streams use the default ILM policy. The retention durations are aligned, and the default ILM policy does not use any advanced functionality (e.g. hot/warm/cold tiering), so this should be considered a non-breaking change.

[pilsy]

It's not really an enhancement, it's more of a bug -- how do i migrate this myself? none of the tracing stuff works anymore in our instances... (we are spending 25k a year on them, and they no longer work)

[axw]

@pilsy which part's a bug? There's definitely a bug in 8.15.0, for which there's a fix in 8.15.1: #13898

Is there something else affecting you too?