Replicate or relocate data via snapshot

[dakrone]

In order to reduce DTS costs for cross-zone data transfer, we should investigate whether we want to replicate or relocate data using a snapshot.

This is close to what a full_copy searchable snapshot index is. Rather than transferring data between ES nodes, we could use an object store as a "free" transferal medium.

Note that since this needs to go through an object store, the index would have to be marked as read-only to prevent data being lost.

There are two options for this, the first being using a regular snapshot and partially restoring it on the receiving node. The second is not using any snapshot infrastructure and instead using S3 as a "temporary staging" ground for relocating the index. Either option will require work, however, as we don't have a way of targeting the restoration of a single shard, and we will need to be able to treat the restoration similar to a regular relocation.

Fully cached searchable snapshot-backed indices do this already, where the recovery source for relocation becomes the snapshot rather than peer recovery, so this is a formalization of that process on a wider scale.

In order for this to be applicable automatically however, it would be useful to implement the concept of a default repository (#66040), so that a user does not need to specify a repository for their index.

If we implemented this using snapshots, we also need to decide whether the snapshot should be a one-off, where a snapshot is taken on-demand for the index, the relocation happens, then the snapshot is removed, or if we can implement it using existing periodic snapshots. We could also use the clone snapshot API to clone index-specific snapshots out of a particular SLM snapshot rather than creating a new one on-demand.

Phase 1

In this phase we will try to use the available snapshots for read-only and low write indices and thus reduce the inter AZ traffic when possible.

• Add cluster setting to enable snapshot based peer recoveries
• Add service to retrieve snapshots information (files info, commit user data, etc) (Add the ability to fetch the latest successful shard snapshot #75080)
• Implement comparison mechanism between snapshot data / recovery source to determine if a snapshot can be used during peer recovery (Add peer recovery planners that take into account available snapshots #75840)
• Add index recovery phase to download snapshot files from repository computed in the previous step (including failure handling) (Add peer recoveries using snapshot files when possible #76237)
• Enhance recovery stats to include information about the data downloaded from the snapshot (Keep track of data recovered from snapshots in RecoveryState #76499)

Phase 2

• Handle recoveries after primary failovers using a sequence numbers (Add support for peer recoveries using snapshots after primary failovers #77420)
• Limit concurrent snapshot file restores in recovery per node #79316
• Ensure proper license is used, recovering from source otherwise (Move SnapshotsRecoveryPlannerService to its own x-pack plugin #79637)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[henrikno]

Another scenario this can help with is scaling out search-heavy clusters. For instance if you have 5 nodes answering search requests for a hot index and they're close to be at capacity. Trying to increase the replicas often makes this worse because now the already busy nodes have to also replicate their shards to new nodes, which might make it take a long time or not be possible because they're oversaturated. Replicating the shards from snapshot storage would be able to handle this without involving the existing hot nodes.