[Remove] Legacy Version support from Snapshot/Restore Service #4728

nknize · 2022-10-10T19:16:26Z

Removes all stale snapshot / restore code for legacy versions prior to OpenSearch 2.0. This includes
handling null ShardGenerations, partial concurrency, null index generations, etc. which are no longer
supported in snapshot versions after legacy 7.5.

relates #2747
depends on #4729

github-actions · 2022-10-10T19:36:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4303/
CommitID: 269b35d

github-actions · 2022-10-10T19:44:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4304/
CommitID: 75119cf

nknize · 2022-10-10T20:57:28Z

Failure is fixed in BWC PR #4729. Updated the description to include dependency.

github-actions · 2022-10-11T17:58:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4421/
CommitID: 75119cf

github-actions · 2022-10-12T04:49:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4467/
CommitID: 3fc04a5

github-actions · 2022-10-12T05:16:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4468/
CommitID: f6b96e2

nknize · 2022-10-12T16:19:10Z

Reproducible failure unrelated to this PR, exists on main as well. Looks like we ignored these failures because they were flakey for so long, #4162. We'll need to track down the introduction separately:

/gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=926BB9577ACEFB6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PR -Dtests.timezone=America/Virgin -Druntime.java=17

Mismatching shard routings: [[gcbrbsekqi][4], node[CrwBliCDQMGz9KGNyAR9HA], [P], s[STARTED], a[id=9Up1aDjPSy6oha_NNMmmzQ], [gcbrbsekqi][5], node[CrwBliCDQMGz9KGNyAR9HA], [P], s[STARTED], a[id=8tAYnIv3TXi-1uJcCFszmg]]
Expected: a collection with size <1>
     but: collection size was <2>

github-actions · 2022-10-12T16:35:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4492/
CommitID: 8da1f30

Removes all stale snapshot / restore code for legacy versions prior to OpenSearch 2.0. This includes handling null ShardGenerations, partial concurrency, null index generations, etc. which are no longer supported in snapshot versions after legacy 7.5. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

github-actions · 2022-10-19T02:27:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/4905/
CommitID: 4282407

…arch-project#4728) Removes all stale snapshot / restore code for legacy versions prior to OpenSearch 2.0. This includes handling null ShardGenerations, partial concurrency, null index generations, etc. which are no longer supported in snapshot versions after legacy 7.5. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x Signed-off-by: Kartik Ganesh <gkart@amazon.com>

This reverts changes made in opensearch-project#4728 and opensearch-project#4702. These were only made in main and not backported to 2.x This change also adds unit tests for IndexMetadataGenerations Signed-off-by: Kartik Ganesh <gkart@amazon.com>

@mch2

…shots (#5812) * Enhance searchable snapshots to enable a read-only view of older snapshots (#5429) * Enhance searchable snapshots to enable a read-only view of older snapshots This change removes the guardrails around N-1 backward compatibility and uses Lucene's "expert" APIs to read snapshots (Lucene segments) older than N-1 on a best-effort basis. The functionality is gated by an additional feature flag, separate from the searchable snapshots flag. Note that the Lucene integration is rather inefficient because the necessary "expert" Lucene APIs are still package-private. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added some unit tests This change also includes a test index ZIP file for the unit tests. The change also introduces a bug fix in the readAnySegmentsInfo method to close the reader before returning the SegmentInfos object - this avoids dangling/open file handles. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporate PR comments from andrross Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove use of IndexSetting for minimum version for snapshots backwards compatibility Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Moved ES 6.3.0 test data to a subdirectory This change also includes an update to the file name to clarify that it is an ES index, and changing the associated markdown file name to just README.md. All tests that reference this ZIP file have corresponding changes to the path they reference. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Update unit tests to use try-with-resources Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Added FeatureFlagSetter helper class Also refactored unit test classes to use the helper class. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Incorporating PR feedback from @mch2 Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix IndexSettingsTests Updated the asserts in IndexSettingsTests to account for the new defaulting behavior. Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixed compile issues after cherry-pick Note that the unit tests are still failing at this commit since the Lucene 9 libraries no longer hold constants for Lucene 7 and below, so the fromId logic resolves the Lucene version to 8. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fix multiple aspects of version resolution This change fixes resolution of the Lucene version for legacy versions since the Lucene 9 libraries no longer hold constants for Lucene 7 and below. The change also updates DECLARED_VERSIONS to derive from the Versions class rather than LegacyESVersions (thereby ignoring legacy versions). This in turn required a change to the minimumIndexCompatibleVersion logic for LegacyESVersion. Finally, the testMinimumIndexCompatibilityVersion unit test was updated to use accurate version identifiers. All unit tests pass and the code compiles, but actual functionality is still broken because some backwards compatibility logic was removed in the current branch that is retained in 2.x Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Reintroducing backwards compatibility logic in certain classes This reverts changes made in #4728 and #4702. These were only made in main and not backported to 2.x This change also adds unit tests for IndexMetadataGenerations Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Declared LegacyESVersion constants for better readability This commit also includes a correction to documentation, and removes the unnecessary "afterWriteSnapBlob" runnable from BlobStoreRepository Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Fixing PR reference in CHANGELOG Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Revert CHANGELOG update This feature was released in 2.5.0 so it no longer needs to be listed in the changelog. Signed-off-by: Kartik Ganesh <gkart@amazon.com> * Remove outdated comment Signed-off-by: Kartik Ganesh <gkart@amazon.com> Signed-off-by: Kartik Ganesh <gkart@amazon.com>

nknize added non-issue bugs / unexpected behaviors that end up non issues; audit trail simple changes that aren't issues >upgrade Label used when upgrading library dependencies (e.g., Lucene) v3.0.0 Issues and PRs related to version 3.0.0 labels Oct 10, 2022

nknize requested review from a team and reta as code owners October 10, 2022 19:16

reta approved these changes Oct 11, 2022

View reviewed changes

nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from 3fc04a5 to f6b96e2 Compare October 12, 2022 04:42

nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from f6b96e2 to 8da1f30 Compare October 12, 2022 16:20

nknize added 3 commits October 18, 2022 20:51

add changelog

504506e

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

add back min_version for now

4282407

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize force-pushed the remove/ancientSnapshotRestoreLogic branch from 8da1f30 to 4282407 Compare October 19, 2022 01:51

nknize merged commit ec34737 into opensearch-project:main Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remove] Legacy Version support from Snapshot/Restore Service #4728

[Remove] Legacy Version support from Snapshot/Restore Service #4728

nknize commented Oct 10, 2022 •

edited

Loading

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

nknize commented Oct 10, 2022

github-actions bot commented Oct 11, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

nknize commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

github-actions bot commented Oct 19, 2022

[Remove] Legacy Version support from Snapshot/Restore Service #4728

[Remove] Legacy Version support from Snapshot/Restore Service #4728

Conversation

nknize commented Oct 10, 2022 • edited Loading

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

nknize commented Oct 10, 2022

github-actions bot commented Oct 11, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 12, 2022

Gradle Check (Jenkins) Run Completed with:

nknize commented Oct 12, 2022

github-actions bot commented Oct 12, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 19, 2022

Gradle Check (Jenkins) Run Completed with:

nknize commented Oct 10, 2022 •

edited

Loading