Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle shard allocation for NONE migration direction #13322

Merged
merged 8 commits into from
May 2, 2024

Conversation

ltaragi
Copy link
Contributor

@ltaragi ltaragi commented Apr 22, 2024

Description

  • Before migration starts, new nodes join the cluster in MIXED compatibility mode, while the migration direction remains at its default value of NONE.
  • We should handle shard allocation correctly under these conditions, based on the index settings.
  • This change handles the following scenarios in RemoteStoreMigrationAllocationDecider.java:
    • For remote store backed indices, shards should only be allocated to remote store nodes.
    • For non remote store backed indices, shards should only be allocated to non-remote store or docrep nodes.
  • Extends validation for TranslogDurabilitySettings to the case of remote store migration by adding extra condition of isMigratingToRemoteStore(clusterSettings) == false here

Related Issues

Resolves #13321

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Apr 22, 2024
Copy link
Contributor

❌ Gradle check result for 9a87afe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ltaragi ltaragi self-assigned this Apr 23, 2024
Copy link
Contributor

✅ Gradle check result for e361cf1: SUCCESS

Copy link

codecov bot commented Apr 23, 2024

Codecov Report

Attention: Patch coverage is 64.70588% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 70.81%. Comparing base (b15cb0c) to head (54bfe73).
Report is 245 commits behind head on main.

Files Patch % Lines
...decider/RemoteStoreMigrationAllocationDecider.java 62.50% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #13322      +/-   ##
============================================
- Coverage     71.42%   70.81%   -0.61%     
- Complexity    59978    60325     +347     
============================================
  Files          4985     5050      +65     
  Lines        282275   286807    +4532     
  Branches      40946    41553     +607     
============================================
+ Hits         201603   203111    +1508     
- Misses        63999    66739    +2740     
- Partials      16673    16957     +284     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ltaragi ltaragi force-pushed the none-direction-allocation branch 2 times, most recently from 9eddae3 to 82c963f Compare April 25, 2024 12:09
Copy link
Contributor

❕ Gradle check result for 82c963f: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ltaragi ltaragi force-pushed the none-direction-allocation branch 2 times, most recently from 9033742 to 8a5ef58 Compare April 30, 2024 06:19
Copy link
Contributor

❌ Gradle check result for 9033742: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 8a5ef58: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Copy link
Contributor

❕ Gradle check result for 54bfe73: UNSTABLE

  • TEST FAILURES:
      2 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod
      1 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ltaragi
Copy link
Contributor Author

ltaragi commented Apr 30, 2024

❕ Gradle check result for 54bfe73: UNSTABLE

  • TEST FAILURES:
      2 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod
      1 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky test #11979

@gbbafna gbbafna merged commit ef841dd into opensearch-project:main May 2, 2024
27 of 28 checks passed
@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label May 2, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 2, 2024
…on (#13322)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
(cherry picked from commit ef841dd)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@gbbafna gbbafna mentioned this pull request May 2, 2024
21 tasks
gbbafna pushed a commit that referenced this pull request May 2, 2024
…on (#13322) (#13501)

(cherry picked from commit ef841dd)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
finnegancarroll pushed a commit to finnegancarroll/OpenSearch that referenced this pull request May 10, 2024
…on (opensearch-project#13322)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request May 17, 2024
…on (opensearch-project#13322)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
parv0201 pushed a commit to parv0201/OpenSearch that referenced this pull request Jun 10, 2024
…on (opensearch-project#13322)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Durability Issues and PRs related to the durability framework Storage:Remote
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[Remote Store] Handle shard allocation for NONE migration direction
3 participants