Allow some Repository Settings to be Updated Dynamically #72543
Conversation
This commit serves two purposes. For one, we need the ability to dynamically update a repository setting for the encrypted repository work. Also, this allows dynamically updating repository rate limits while snapshots are in progress. This has often been an issue in the past where a long running snapshot made progress over a long period of time already but is going too slowly with the current rate limit. This left no good options, either throw away the existing partly done snapshot's work and recreate the repo with a higher rate limit to speed things up or wait for a long time with the current rate limit. With this change the rate limit can simply be increased while a snapshot or restore is running and will take effect imidiately.
original-brownbear
added
>enhancement
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
server/src/main/java/org/elasticsearch/repositories/RepositoriesService.java
Outdated
Show resolved
Hide resolved
| @@ -469,6 +487,15 @@ public void applyClusterState(ClusterChangedEvent event) { | |||
| } | |||
| } | |||
|
|
|||
| private boolean canUpdateInPlace(DiscoveryNodes nodes, RepositoryMetadata updatedMetadata, Repository repository) { | |||
| if (nodes.getMinNodeVersion().before(UPDATE_IN_PLACE_VERSION)) { | |||
There was a problem hiding this comment.
This looks fragile because the minimum node version in the cluster can go downwards (if UPDATE_IN_PLACE_VERSION doesn't ultimately end in .0.0). I think it's ok if we're checking this as part of a cluster state update, because a node won't be concurrently rejoining the cluster. Can we assert we're on the master thread at this point to make sure we don't check this wrongly?
There was a problem hiding this comment.
Actually why do we need the version check? If we update the repository metadata and there's older nodes around then will they break or will they just keep using the old repo metadata for any ongoing shard-level operations?
There was a problem hiding this comment.
Alrighty, as just discussed. The BwC will be handled via some other mechanism here. Removed the BwC logic from this PR entirely for now as it's not a problem in interaction with versions that aren't already a problem anyway.
|
@DaveCTurner ping if you have some time for this one :) Would be nice to get this in at some point to unblock some of the encrypted repository work. I think the BwC situation isn't much of a concern here after thinking it through. In the end all it will do is fail shard snapshots on older data nodes, but I don't think there is any potential for repo-corruption even when working with pre-7.6 nodes. |
server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
Outdated
Show resolved
Hide resolved
|
Thanks David! |
This commit serves two purposes. For one, we need the ability to dynamically update a repository setting for the encrypted repository work. Also, this allows dynamically updating repository rate limits while snapshots are in progress. This has often been an issue in the past where a long running snapshot made progress over a long period of time already but is going too slowly with the current rate limit. This left no good options, either throw away the existing partly done snapshot's work and recreate the repo with a higher rate limit to speed things up or wait for a long time with the current rate limit. With this change the rate limit can simply be increased while a snapshot or restore is running and will take effect imidiately.
…4052) This commit serves two purposes. For one, we need the ability to dynamically update a repository setting for the encrypted repository work. Also, this allows dynamically updating repository rate limits while snapshots are in progress. This has often been an issue in the past where a long running snapshot made progress over a long period of time already but is going too slowly with the current rate limit. This left no good options, either throw away the existing partly done snapshot's work and recreate the repo with a higher rate limit to speed things up or wait for a long time with the current rate limit. With this change the rate limit can simply be increased while a snapshot or restore is running and will take effect immediately.
This commit serves two purposes. For one, we need the ability to dynamically
update a repository setting for the encrypted repository work (hence the ignored settings parameter enabling
nested repo types).
Also, this allows dynamically updating repository rate limits while snapshots are
in progress. This has often been an issue in the past where a long running snapshot
made progress over a long period of time already but is going too slowly with the
current rate limit. This left no good options, either throw away the existing
partly done snapshot's work and recreate the repository with a higher rate limit to speed
things up or wait for a long time with the current rate limit.
With this change the rate limit can simply be increased while a snapshot or restore
is running and will take effect immediately.
cc @albertzaharovits