[Request] Document how to retain data with fleet integration and ILM policies

[paulb-elastic]

Description

This is the observability-docs issue for elastic/synthetics#286.

When users take advantage of configuring Synthetic (real browser) monitors through a Fleet Integration and Elastic Agent (as opposed to a heartbeat.yml manually deployed), this will use data streams. This implements separate indices for screenshots, network and everything else, meaning users can have a different (probably more aggressive) retention policy for screenshots and even network data.

As screenshots can quickly consume a fair amount of space and start to cost more, you could imagine for example a 2 week retention for screenshots, a 6 month retention for network data (used to generate the waterfall charts) and say 24 months the main result data.

This needs to be documented so users can take advantage of this (we won't be implementing default ILM policies, or other UI controls for configuring this, at this stage).

Collaboration

• Other - needs input from the Synthetics team

Suggested Target Release

We are expecting to release Synthetic (real browser) and Elastic Agent Integration in 7.15.

Stakeholders

• @paulb-elastic
• @andrewvc
• @vigneshshanmugam
• @drewpost

[colleenmcginnis]

👋 @paulb-elastic is this a good time for me to start working on this issue or should I wait per elastic/synthetics#286 (comment)?

[paulb-elastic]

@colleenmcginnis yes, I'd suggest holding off for now, as we need to prioritise the work to simplify the ILM policy piece (we’ve made progress with a good level of detail in elastic/uptime#453 (comment) but this capability isn’t quite available yet, which will make it much easier to use, and set specific policies for each data type).

[paulb-elastic]

Updating with the implementation issue for the additional capability: elastic/uptime#462
(not worth updating the docs until this is merged)

[colleenmcginnis]

@paulb-elastic is there anything else we need to add to the docs related to this topic that isn't already covered in https://www.elastic.co/guide/en/observability/current/synthetics-manage-retention.html?

[paulb-elastic]

Thanks @colleenmcginnis you are right, it was done in #1944