ILM rollover within the same availability zone

[obogobo]

Here's an example for an index with one replica, while using force_awareness node attributes to spread the shards across 2x AZ's via the zone property. Right now, indexing / ILM are implemented as such:

Indexing:

• online replication - each doc is sent from the receiving (gateway) node to another instance in a separate AZ, to be indexed in parallel with its primary shard

ILM:

• eventually an ILM trigger condition is met
• a shard recovery from hot -> warm
• another shard recovery from warm -> warm
• delete the source shards

This means there are up to 3x network boundary transitions for the lifecycle of a document 🙀
And you better believe Amazon's making hay while the sun's shining at $0.01 / GB transferred across zones.

A solution to this might be an added config option to prefer replication on rollover within the same availability zone (where data transfer is free), according to: https://www.elastic.co/guide/en/elasticsearch/reference/7.8/modules-cluster.html#forced-awareness

i.e., ILM changes the index's routing.allocation.require attribute to start the rollover process, followed by the primary and replica shards getting bulk transferred in parallel to nodes with the new attribute, attempting to pick destinations in the same zone they originated from.

Unless something like this is already possible, and if so I'd love to hear about it! Thanks for the consideration!

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[seang-es]

This is not just applicable to ILM, users have raised questions about this in the case where you might be patching a node in one AZ and spinning up another node to take its place. Copies will be coming from the primary shard, whichever zone that might be in, rather than the still available replica in the local zone.

[DaveCTurner]

Relates #63519.

[joegallo]

I'm removing the team-discuss label from some older Team:Data Management issues -- we've had plenty of time to discuss them, but we haven't, so the label isn't serving its purpose. Feel free to delete this comment and/or re-add the team-discuss label.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[DaveCTurner]

I think this duplicates the (resolved) #73496: data transfer to/from snapshots does not incur cross-zone network traffic costs. Therefore I'm closing it.

[obogobo]

is it possible to do ILM rollovers by snapshotting to S3 from a hot node and then recovering on a warm node? instead of a node to node direct data transfer? if so that'd be amazing, and this issue relates - otherwise, not sure that one duplicates this issue.

[DaveCTurner]

Yes that's right, that's what #73496 does.