Integrate local recovery with remote store seeding during migration #12922

BhumikaSaini-Amazon · 2024-03-26T07:48:15Z

Description

Integrates local recovery of shard with remote store seeding during migration of cluster to a remote backed cluster. This ensures that the data is uploaded to the remote store after performing a rolling restart as part of the docrep to remote store migration. Node restarts will also check if seeding is required and based on it will upload the data to remote.

Related Issues

#12246
#12718

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~ (will be covered under changelog entry for the high-level feature support)
~~Public documentation issue/PR created~~ (not applicable)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-03-26T08:12:43Z

Compatibility status:

Checks if related components are compatible with change 5c328d5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-03-26T08:34:38Z

❌ Gradle check result for 59cd374: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-01T05:28:10Z

❌ Gradle check result for 86ed7cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...rc/internalClusterTest/java/org/opensearch/remotemigration/RemotePrimaryLocalRecoveryIT.java

github-actions · 2024-04-09T10:42:05Z

❌ Gradle check result for 44e7217: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-09T11:00:04Z

❌ Gradle check result for f8339bc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-09T12:04:24Z

❕ Gradle check result for 0afcdd2: UNSTABLE

TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-04-09T12:05:46Z

Codecov Report

Attention: Patch coverage is 52.17391% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 71.34%. Comparing base (b15cb0c) to head (d28f6da).
Report is 218 commits behind head on main.

Files	Patch %	Lines
...java/org/opensearch/index/shard/StoreRecovery.java	0.00%	6 Missing and 1 partial ⚠️
...rg/opensearch/index/translog/RemoteFsTranslog.java	66.66%	1 Missing and 1 partial ⚠️
...c/main/java/org/opensearch/index/IndexService.java	0.00%	1 Missing ⚠️
...org/opensearch/index/seqno/ReplicationTracker.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12922      +/-   ##
============================================
- Coverage     71.42%   71.34%   -0.08%     
- Complexity    59978    60653     +675     
============================================
  Files          4985     5044      +59     
  Lines        282275   285711    +3436     
  Branches      40946    41373     +427     
============================================
+ Hits         201603   203833    +2230     
- Misses        63999    65022    +1023     
- Partials      16673    16856     +183

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

...rc/internalClusterTest/java/org/opensearch/remotemigration/RemotePrimaryLocalRecoveryIT.java

server/src/main/java/org/opensearch/index/IndexService.java

...rc/internalClusterTest/java/org/opensearch/remotemigration/RemotePrimaryLocalRecoveryIT.java

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

github-actions · 2024-04-25T05:53:47Z

❕ Gradle check result for 9be0001: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

BhumikaSaini-Amazon · 2024-04-25T05:57:10Z

org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock => reported flaky in #10006

github-actions · 2024-04-25T06:18:10Z

❕ Gradle check result for d28f6da: UNSTABLE

TEST FAILURES:

      2 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2024-04-25T06:58:32Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12922-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a8008e2f37ec7dca061fcb7b5f84c944dd4f1108
# Push it to GitHub
git push --set-upstream origin backport/backport-12922-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12922-to-2.x.

…pensearch-project#12922) Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

…12922) Signed-off-by: Bhumika Saini <sabhumik@amazon.com> (cherry picked from commit a8008e2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…12922) (#13428) (cherry picked from commit a8008e2) Signed-off-by: Bhumika Saini <sabhumik@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

BhumikaSaini-Amazon force-pushed the mixed-local-recovery-2 branch from 59cd374 to 86ed7cb Compare April 1, 2024 05:15

gbbafna reviewed Apr 5, 2024

View reviewed changes

...rc/internalClusterTest/java/org/opensearch/remotemigration/RemotePrimaryLocalRecoveryIT.java Outdated Show resolved Hide resolved

gbbafna reviewed Apr 5, 2024

View reviewed changes

...rc/internalClusterTest/java/org/opensearch/remotemigration/RemotePrimaryLocalRecoveryIT.java Outdated Show resolved Hide resolved

BhumikaSaini-Amazon force-pushed the mixed-local-recovery-2 branch from 86ed7cb to 44e7217 Compare April 9, 2024 10:34

BhumikaSaini-Amazon force-pushed the mixed-local-recovery-2 branch 2 times, most recently from f8339bc to 0afcdd2 Compare April 9, 2024 10:59

gbbafna reviewed Apr 9, 2024

View reviewed changes

BhumikaSaini-Amazon force-pushed the mixed-local-recovery-2 branch from 0afcdd2 to 5c328d5 Compare April 9, 2024 14:10

BhumikaSaini-Amazon marked this pull request as ready for review April 9, 2024 14:13

BhumikaSaini-Amazon requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, kotwanikunal, mch2, msfroh, nknize and owaiskazi19 as code owners April 9, 2024 14:13

Bhumika Saini added 11 commits April 25, 2024 10:21

Integrate local recovery with remote store seeding during migration

6d697cb

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Empty commit to retrigger build

12df6fd

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Retain local translog during seeding

6250147

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Apply spotless fixes

92b6460

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Fix misconfigured replicas

297223d

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Add seedRemote flag to TranslogConfig

5d5c2bd

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Empty commit to retrigger build

dc345ed

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Address PR comments

28a6e78

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Ensure graceful test cleanup, Fix assertion in ReplicationTracker

f379001

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Fix ReplicationTrackerTests

2b48295

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Use newVersion for ReplicationTracker assertion during migration

9be0001

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

BhumikaSaini-Amazon force-pushed the mixed-local-recovery-2 branch from 5505cfa to 9be0001 Compare April 25, 2024 05:03

shourya035 reviewed Apr 25, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/seqno/ReplicationTracker.java Outdated Show resolved Hide resolved

Remove redundant null check

d28f6da

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

shourya035 approved these changes Apr 25, 2024

View reviewed changes

gbbafna merged commit a8008e2 into opensearch-project:main Apr 25, 2024
28 of 29 checks passed

gbbafna added the backport 2.x Backport to 2.x branch label Apr 25, 2024

opensearch-trigger-bot bot added the backport-failed label Apr 25, 2024

BhumikaSaini-Amazon pushed a commit to BhumikaSaini-Amazon/OpenSearch that referenced this pull request Apr 25, 2024

Integrate local recovery with remote store seeding during migration (o…

e5de59b

…pensearch-project#12922) Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

BhumikaSaini-Amazon mentioned this pull request Apr 25, 2024

Integrate local recovery with remote store seeding during migration #13376

Closed

8 tasks

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch backport-failed labels Apr 28, 2024

opensearch-trigger-bot bot mentioned this pull request Apr 28, 2024

[Backport 2.x] Integrate local recovery with remote store seeding during migration #13428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate local recovery with remote store seeding during migration #12922

Integrate local recovery with remote store seeding during migration #12922

BhumikaSaini-Amazon commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

codecov bot commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 25, 2024

BhumikaSaini-Amazon commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

opensearch-trigger-bot bot commented Apr 25, 2024

Integrate local recovery with remote store seeding during migration #12922

Integrate local recovery with remote store seeding during migration #12922

Conversation

BhumikaSaini-Amazon commented Mar 26, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Mar 26, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Mar 26, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

codecov bot commented Apr 9, 2024 • edited Loading

Codecov Report

github-actions bot commented Apr 25, 2024

BhumikaSaini-Amazon commented Apr 25, 2024

github-actions bot commented Apr 25, 2024

opensearch-trigger-bot bot commented Apr 25, 2024

BhumikaSaini-Amazon commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

codecov bot commented Apr 9, 2024 •

edited

Loading