make pushes concurrent and limit pull concurrency #72
Conversation
Previously, pushes were serialized and pulls were unboundedly concurrent. This makes pushes concurrent (capped at 16 concurrent uploads) and also caps download concurrency at 16. Signed-off-by: Joel Dice <joel.dice@fermyon.com>
dicej
requested review from
thomastaylor312,
bacongobbler and
flavio
as code owners
|
Thanks @dicej! Do you think it'd be useful if the maximum number of concurrent pulls/pushes was dependent on the system hardware? For example, a Raspberry Pi may want to serialize less concurrent pulls/pushes, while a more beefy dev workstation may want to increase that count. Most programs make this configurable by reading the number of available CPUs and provide an option to modify the default in case there are specific use cases (throttling/benchmarking). What do you think about that approach? |
|
@bacongobbler Given that uploads and downloads are generally I/O limited rather than CPU-limited, I don't think the number of CPU cores is an appropriate metric to use. For example, a Raspberry Pi on a gigabit internet connection is likely to perform better than a 12 core laptop tethered to a slow cellphone connection. I do agree that adjusting the number based on some kind of runtime detection would be great; just not sure what that detection would look like (or whether it would be worthwhile). |
|
Hmm. Perhaps it could be an optional field in the |
|
I wonder if we need to make two separate options: one for uploads ( |
|
Yeah, that sounds great. Update coming shortly. |
This introduces `pull_with_max_concurrency` and `push_with_max_concurrency`, allowing the application to specify download and upload concurrency limits, respectively. Signed-off-by: Joel Dice <joel.dice@fermyon.com>
|
Overall ok, but I would have preferred to have these configuration knobs added to the |
|
Ah, good point. I misread @bacongobbler's comment. I'll address that when I'm back at my keyboard. |
...and use them to determine maximum upload and download concurrency for pushes and pulls, respectively. I've also bumped the version up to 0.10.0 since this is a breaking API change (due to new fields being added to a public struct composed of all-public fields). Signed-off-by: Joel Dice <joel.dice@fermyon.com>
|
I've pushed an update. Note that this required a version bump due to a breaking API change (new fields on the |
What's Changed: * client: Fix failed to pull ubuntu image from docker.io by @arronwy in oras-project#67 * Update rstest requirement from 0.16.0 to 0.17.0 by @dependabot in oras-project#70 * Bump actions/checkout from 3.3.0 to 3.4.0 by @dependabot in oras-project#69 * Bump actions/checkout from 3.4.0 to 3.5.0 by @dependabot in oras-project#73 * Change the visibility of `OciEnvelop.errors` to `pub` by @linyinfeng in oras-project#71 * Bump actions/checkout from 3.5.0 to 3.5.2 by @dependabot in oras-project#74 * make pushes concurrent and limit pull concurrency by @dicej in oras-project#72 * Update default `docker.io` registry by @radu-matei in oras-project#81 * Fix: Expose HTTP errors when pulling layers by @jsoverson in oras-project#82 * client: Return Stream for async_pull_blob API by @arronwy in oras-project#83 * chore(deps): Bump actions/checkout from 3.5.2 to 3.5.3 by @dependabot in oras-project#85 * fix(CI): address warnings of cargo deny by @flavio in oras-project#84 * consider ARCH as ppc64le when rust arch is powerpc64 by @Amulyam24 in oras-project#86 * chore(deps): Update rstest requirement from 0.17.0 to 0.18.1 by @dependabot in oras-project#88 * add optional artifactType to image manifest for oci v1.1 by @devigned in oras-project#89 * Higher-ranked lifetime error workaround by @linyinfeng in oras-project#90 * chore(deps): Bump actions/checkout from 3.5.3 to 3.6.0 by @dependabot in oras-project#91 * chore(deps): Bump actions/checkout from 3.6.0 to 4.0.0 by @dependabot in oras-project#92 * chore(deps): Bump actions/checkout from 4.0.0 to 4.1.0 by @dependabot in oras-project#94 * Improved Push Logic by @rbbl-dev in oras-project#93 * Update the configfile to match the oci v1 spec by @lswith in oras-project#77 * Conversion between a Config and a ConfigFile by @lswith in oras-project#76 * Enhance client to push blobs, mount blobs, and push raw manifests by @aochagavia in oras-project#66 * fix: bring back workaround for rustc by @flavio in oras-project#95 * implemented method to list tags by @rbbl-dev in oras-project#80 * feat(config.rs): add OS type for WASI Preview 1 by @vdice in oras-project#99 * chore(deps): Bump actions/checkout from 4.1.0 to 4.1.1 by @dependabot in oras-project#100 * chore(deps): Update testcontainers requirement from 0.14 to 0.15 by @dependabot in oras-project#96 * feat: return auth token from Client::auth by @mattarnoatibm in oras-project#105 Signed-off-by: Flavio Castelli <fcastelli@suse.com>
Previously, pushes were serialized and pulls were unboundedly concurrent. This makes pushes concurrent (capped at 16 concurrent uploads) and also caps download concurrency at 16.