Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Explore parallel upload of segment files #7385

Closed
linuxpi opened this issue May 3, 2023 · 1 comment
Closed

[Remote Store] Explore parallel upload of segment files #7385

linuxpi opened this issue May 3, 2023 · 1 comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request

Comments

@linuxpi
Copy link
Collaborator

linuxpi commented May 3, 2023

Is your feature request related to a problem? Please describe.
As part of Remote store segment upload flow, we upload multiple files to remote store directory. Today this upload is sequential and no two files are uploaded parallel. Which could be increasing latency and impacting overall throughput.

Describe the solution you'd like
Explore if parallelization of segment files upload can help improve latency and throughout. Although we need to be mindful to not bombard s3 with requests get throttled. Run performance tests to figure out baseline limits for a cluster/node after this change

Describe alternatives you've considered
NA

Additional context
NA

@linuxpi linuxpi added enhancement Enhancement or improvement to existing feature or request untriaged labels May 3, 2023
@linuxpi linuxpi closed this as completed Jun 14, 2023
@linuxpi
Copy link
Collaborator Author

linuxpi commented Jun 14, 2023

not needed after #6703

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
distributed framework enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

2 participants