Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SnapshotV2] Support centralize snapshot creation #15124

Merged
merged 35 commits into from
Aug 28, 2024

Conversation

anshu1106
Copy link
Contributor

@anshu1106 anshu1106 commented Aug 6, 2024

Description

This PR centralizes snapshot create operations and integrate with timestamp pinning for snapshot creation. The snapshot creation has a sync execution and it can be scaled independently of number of shards in the cluster.
More details can be found in the RFC

Related Issues

Resolves #15123

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ocking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Snapshots labels Aug 6, 2024
Copy link
Contributor

github-actions bot commented Aug 6, 2024

❌ Gradle check result for 9142a25: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

github-actions bot commented Aug 6, 2024

❌ Gradle check result for d9bbc65: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Anshu Agarwal added 2 commits August 6, 2024 16:53
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

github-actions bot commented Aug 6, 2024

❌ Gradle check result for 8e85231: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@Bukhtawar Bukhtawar changed the title [Shallow SnapshotV2] Support centralize snapshot creation [SnapshotV2] Support centralize snapshot creation Aug 6, 2024
Copy link
Contributor

github-actions bot commented Aug 7, 2024

❌ Gradle check result for 8a507ad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

github-actions bot commented Aug 7, 2024

❌ Gradle check result for 2d404e8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

github-actions bot commented Aug 8, 2024

❌ Gradle check result for 90c860c: TIMEOUT

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Copy link
Contributor

❌ Gradle check result for 193da65: TIMEOUT

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
@anshu1106 anshu1106 marked this pull request as ready for review August 14, 2024 11:36
Copy link
Contributor

❕ Gradle check result for fb48a2d: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ashking94 ashking94 merged commit 23cba28 into opensearch-project:main Aug 28, 2024
33 of 34 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15124-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 23cba282bbb3276781a71ed3ace315abb90f07d2
# Push it to GitHub
git push --set-upstream origin backport/backport-15124-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15124-to-2.x.

anshu1106 added a commit to anshu1106/OpenSearch that referenced this pull request Sep 2, 2024
…#15124)

* Initial Commit to support centralize snapshot creation and implicit locking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix deserilization error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix gradle spotless check

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix listener

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix test

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix snapshot generation

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify cluster setting name

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Uncomment pin timestamp code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify log messages

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add spotless check failure fix

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix completion listener for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Elevate cluster state update priority for repository metadata update task

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add priority as IMMEDIATE for cluster state repo update task only for v2 snapshots

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix build error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add repository setting for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add integ test to verify snapshot creation if shallow copy repo setting is disabled

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless vilation error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comment

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add min version check for backward compatibility

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add integ test for master failover scenario

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* refactor code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add changelog

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add pinned timestamp setting in integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

---------

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 23cba28)
anshu1106 added a commit to anshu1106/OpenSearch that referenced this pull request Sep 2, 2024
…#15124)

* Initial Commit to support centralize snapshot creation and implicit locking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix deserilization error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix gradle spotless check

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix listener

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix test

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix snapshot generation

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify cluster setting name

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Uncomment pin timestamp code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify log messages

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add spotless check failure fix

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix completion listener for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Elevate cluster state update priority for repository metadata update task

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add priority as IMMEDIATE for cluster state repo update task only for v2 snapshots

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix build error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add repository setting for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add integ test to verify snapshot creation if shallow copy repo setting is disabled

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless vilation error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comment

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add min version check for backward compatibility

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add integ test for master failover scenario

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* refactor code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add changelog

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add pinned timestamp setting in integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

---------

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 23cba28)
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
ashking94 pushed a commit that referenced this pull request Sep 3, 2024
* [SnapshotV2] Support centralize snapshot creation (#15124)

* Initial Commit to support centralize snapshot creation and implicit locking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix deserilization error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix gradle spotless check

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix listener

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix test

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix snapshot generation

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify cluster setting name

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Uncomment pin timestamp code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify log messages

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add spotless check failure fix

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix completion listener for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Elevate cluster state update priority for repository metadata update task

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add priority as IMMEDIATE for cluster state repo update task only for v2 snapshots

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix build error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add repository setting for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add integ test to verify snapshot creation if shallow copy repo setting is disabled

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless vilation error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comment

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add min version check for backward compatibility

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add integ test for master failover scenario

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* refactor code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add changelog

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add pinned timestamp setting in integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

---------

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
(cherry picked from commit 23cba28)
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless check failure

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* fix :server:japicmp failure

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* backort PR#15602

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

---------

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Sep 10, 2024
…#15124)

* Initial Commit to support centralize snapshot creation and implicit locking mechanism

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix deserilization error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix gradle spotless check

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix listener

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix test

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix snapshot generation

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify cluster setting name

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Uncomment pin timestamp code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Modify log messages

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add spotless check failure fix

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix completion listener for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Elevate cluster state update priority for repository metadata update task

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add priority as IMMEDIATE for cluster state repo update task only for v2 snapshots

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix build error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add repository setting for snapshot v2

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add integ test to verify snapshot creation if shallow copy repo setting is disabled

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Fix spotless vilation error

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comment

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add min version check for backward compatibility

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* address review comments

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add integ test for master failover scenario

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add more integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* refactor code

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* add changelog

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

* Add pinned timestamp setting in integ tests

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>

---------

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed enhancement Enhancement or improvement to existing feature or request Storage:Snapshots
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

[Snapshot-V2] Centralize snapshot creation and integrate with timestamp pinning
5 participants