Optimize SLM Policy Queries #79341
Merged
Optimize SLM Policy Queries #79341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Same as elastic#79321 but for SLM policies. Enhances RepositoryData accordingly to enable the optimization.
original-brownbear
added
>enhancement
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
DaveCTurner
reviewed
Oct 19, 2021
| @@ -156,7 +156,8 @@ public void testEnforcedCooldownPeriod() throws IOException { | |||
| SnapshotState.SUCCESS, | |||
| SnapshotsService.SHARD_GEN_IN_REPO_DATA_VERSION.minimumCompatibilityVersion(), | |||
| 0L, // -1 would refresh RepositoryData and find the real version | |||
| 0L // -1 would refresh RepositoryData and find the real version | |||
| 0L, // -1 would refresh RepositoryData and find the real version, | |||
| null // null would refresh RepositoryData and find the real version | |||
There was a problem hiding this comment.
The comment suggests that this shouldn't be null.
| @@ -666,7 +666,7 @@ private static SnapshotsInRepo sortSnapshots( | |||
| } | |||
| } | |||
|
|
|||
| private static Predicate<SnapshotInfo> filterBySLMPolicies(String[] slmPolicies) { | |||
| private static Tuple<Predicate<SnapshotInfo>, BiPredicate<SnapshotId, RepositoryData>> filterBySLMPolicies(String[] slmPolicies) { | |||
There was a problem hiding this comment.
Tuple<Predicate<SnapshotInfo>, BiPredicate<SnapshotId, RepositoryData>> is getting a bit convoluted, really it's just the following but without the names or anywhere to hang docs:
interface SomeName {
boolean method1(SnapshotInfo snapshotInfo);
boolean method2(SnapshotId snapshotId, RepositoryData repositoryData);
}
I think the names & docs would be useful, let's make this a thing. Maybe we should be subclassing SnapshotPredicates rather than having these higher-order fields.
There was a problem hiding this comment.
++ I did this for this specific case, now but I agree, it's a worthwhile cleanup all over :) I'll do it in a follow-up after FF
DaveCTurner
approved these changes
Oct 19, 2021
|
Thanks David!
I think that's just fine. The extra bytes in the serialized objects hardly matter and on heap we're deduplicating the strings so they don't matter either. The performance gain when filtering get-snapshots by slm policy on the other hand is quite significant (particularly for the expected case of listing hundreds of backups but maybe also having thousands of ILM created frozen tier snapshots in the repo). |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Oct 19, 2021
Same as elastic#79321 but for SLM policies. Enhances RepositoryData accordingly to enable the optimization.
original-brownbear
added a commit
that referenced
this pull request
Oct 19, 2021
weizijun
added a commit
to weizijun/elasticsearch
that referenced
this pull request
Oct 19, 2021
* upstream/master: Validate tsdb's routing_path (elastic#79384) Adjust the BWC version for the return200ForClusterHealthTimeout field (elastic#79436) API for adding and removing indices from a data stream (elastic#79279) Exposing the ability to log deprecated settings at non-critical level (elastic#79107) Convert operator privilege license object to LicensedFeature (elastic#79407) Mute SnapshotBasedIndexRecoveryIT testSeqNoBasedRecoveryIsUsedAfterPrimaryFailOver (elastic#79456) Create cache files with CREATE_NEW & SPARSE options (elastic#79371) Revert "[ML] Use a new annotations index for future annotations (elastic#79151)" [ML] Use a new annotations index for future annotations (elastic#79151) [ML] Removing legacy code from ML/transform auditor (elastic#79434) Fix rate agg with custom `_doc_count` (elastic#79346) Optimize SLM Policy Queries (elastic#79341) Fix execution of exists query within nested queries on field with doc_values disabled (elastic#78841) Stricter UpdateSettingsRequest parsing on the REST layer (elastic#79227) Do not release snapshot file download permit during recovery retries (elastic#79409) Preserve request headers in a mixed version cluster (elastic#79412) Adjust versions after elastic#79044 backport to 7.x (elastic#79424) Mute BulkByScrollUsesAllScrollDocumentsAfterConflictsIntegTests (elastic#79429) Fail on SSPL licensed x-pack sources (elastic#79348) # Conflicts: # server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Oct 19, 2021
`SnapshotPredicate` and `SnapshotPredicates` do almost the same thing and the way that the `SnapshotPredicates` is constructed is quite convoluted. This commit combines the two classes into one and simplifies the construction logic. Relates elastic#79341
DaveCTurner
added a commit
that referenced
this pull request
Oct 20, 2021
`SnapshotPredicate` and `SnapshotPredicates` do almost the same thing and the way that the `SnapshotPredicates` is constructed is quite convoluted. This commit combines the two classes into one and simplifies the construction logic. Relates #79341
DaveCTurner
added a commit
that referenced
this pull request
Oct 20, 2021
`SnapshotPredicate` and `SnapshotPredicates` do almost the same thing and the way that the `SnapshotPredicates` is constructed is quite convoluted. This commit combines the two classes into one and simplifies the construction logic. Relates #79341
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Same as #79321 but for SLM policies. Enhances RepositoryData accordingly
to enable the optimization.