Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Add multipart upload integration for translog and segment files #7119

Merged

Conversation

raghuvanshraj
Copy link
Contributor

@raghuvanshraj raghuvanshraj commented Apr 12, 2023

Credits: @vikasvb90 and @itiyamas for the design and core implementations of the feature.

Description

Adds multipart parallel uploads in segment and translog upload path for remote store utilizing utilities and models added in #7000.

Issues Resolved

Part of #6703

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

Copy link
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have done high level review, will review once more.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj changed the title Add multipart upload integration for translog and segment files [Remote Store] Add multipart upload integration for translog and segment files May 25, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj
Copy link
Contributor Author

Failure from gradle CI logs:

* What went wrong:
Execution failed for task ':qa:mixed-cluster:v2.8.0#mixedClusterTest'.
> There were failing tests. See the report at: file:///var/jenkins/workspace/gradle-check/search/qa/mixed-cluster/build/reports/tests/v2.8.0%23mixedClusterTest/index.html

This seems to be a flaky test as it is passing when I run it locally. I've opened a bug for the same: #7755

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: 55a31ac
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: c48cc33
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 11, 2023
@raghuvanshraj raghuvanshraj added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jul 11, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 11, 2023
…ent files (#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
(cherry picked from commit 5f98d67)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Jul 12, 2023
---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
vikasvb90 pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
vikasvb90 pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
raghuvanshraj added a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
raghuvanshraj pushed a commit to raghuvanshraj/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
sachinpkale pushed a commit that referenced this pull request Jul 12, 2023
* [Remote Store] Add multipart upload integration for translog and segment files (#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Fix remote segments sync retry regression introduced in PR #7119 (#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>

* Empty-Commit

Signed-off-by: Ashish Singh <ssashish@amazon.com>

---------

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
buddharajusahil pushed a commit to buddharajusahil/OpenSearch that referenced this pull request Jul 18, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: sahil buddharaju <sahilbud@amazon.com>
buddharajusahil pushed a commit to buddharajusahil/OpenSearch that referenced this pull request Jul 18, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: sahil buddharaju <sahilbud@amazon.com>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ent files (opensearch-project#7119)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…-project#7119 (opensearch-project#8632)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants