-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid regular indices in frozen tier #70141
Avoid regular indices in frozen tier #70141
Conversation
The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.
Pinging @elastic/es-distributed (Team:Distributed) |
Pinging @elastic/es-core-features (Team:Core/Features) |
A setting validator can declare settings that the validation depends on, but when updating index settings, we eagerly validate the settings before submitting the cluster state update and here we do not know the existing settings. With this commit, we ensure that the pre-validation works with such validators, letting the validator validate syntax (validate(value)) only. Relates elastic#70141.
if (validateDependencies) { | ||
setting.get(settings); | ||
} else { | ||
setting.validateWithoutDependencies(settings); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change and the associated change in SecureSettings
and tests have been split into #70144
These tests are going to be touched by other work, will wait merging this until that is done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to self to remove AwaitsFix before merge. Those tests are fixed by #70158, which I will integrate here before merging.
Now done.
...de/src/javaRestTest/java/org/elasticsearch/xpack/ilm/actions/SearchableSnapshotActionIT.java
Outdated
Show resolved
Hide resolved
...de/src/javaRestTest/java/org/elasticsearch/xpack/ilm/actions/SearchableSnapshotActionIT.java
Outdated
Show resolved
Hide resolved
...de/src/javaRestTest/java/org/elasticsearch/xpack/ilm/actions/SearchableSnapshotActionIT.java
Outdated
Show resolved
Hide resolved
@elasticmachine run elasticsearch-ci/bwc |
…_not_on_frozen_tier
…_not_on_frozen_tier
Old 7.12/7.13 clusters will create the .snapshot-blob-cache index with tier preference including frozen, which causes cluster state updates to fail when upgrading. Disabling bwc until this is in 7.12.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've just left one small comment, otherwise looking good. I did not review the dependent setting validation logic.
|
||
public static boolean isSearchableSnapshotStore(Settings indexSettings) { | ||
return SNAPSHOT_DIRECTORY_FACTORY_KEY.equals(INDEX_STORE_TYPE_SETTING.get(indexSettings)); | ||
} | ||
|
||
public static boolean isPartialSearchableSnapshotIndex(Map<Setting<?>, Object> indexSettings) { | ||
return SNAPSHOT_DIRECTORY_FACTORY_KEY.equals(indexSettings.get(INDEX_STORE_TYPE_SETTING)) | ||
&& (boolean) indexSettings.get(SNAPSHOT_PARTIAL_SETTING); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this method throw a NPE if indexSettings does not contain SNAPSHOT_PARTIAL_SETTING?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it could, but the expectation is that the map contains the two settings. I have added assertions to enforce this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…_not_on_frozen_tier
…_not_on_frozen_tier
The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.
The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.
Reenable bwc tests now that elastic#70141 has been backported.
This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to elastic#70846, elastic#71013, elastic#70786, elastic#70141
This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to #70846, #71013, #70786, #70141
This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to elastic#70846, elastic#71013, elastic#70786, elastic#70141
…#71129) This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to #70846, #71013, #70786, #70141
The frozen tier will be dedicated for partially cached searchable
snapshots. This PR ensures that we do not allow allocating regular
indices (including fully cached searchable snapshots) to the frozen
tier.