Adjust xz compression settings #123
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Mark-Simulacrum: no appropriate reviewer found, use r? to override |
|
The main tradeoff here is the dictionary size; currently our preset configures it to 8 MB and this raises it to 128 MB. Generally the effect is that we get better compression, but decompression (i.e., rustup) will also require more memory (8 -> 128 MB, roughly, though obviously there are other users of memory there). I think given how much memory running rustc on any reasonable sized crate will take, this seems fine to me. IMO if users are worried about memory usage on this order of magnitude rustup etc are probably not great tools as-is for them regardless, and we should work to enable something much lower impact (e.g., non-compressed downloads or something). |
This adjusts our compression settings for xz compressed files, which should reduce download sizes by around 5-15%, at the cost of ~44% longer compression times. Given that compression happens almost universally in CI, rather than something humans are directly waiting for, this tradeoff feels worth it. If we end up seeing this be too significant an increase in CI times, moving the more aggressive compression solely to promote-release should be relatively straightforward and give most of the benefits.
Mark-Simulacrum
force-pushed
the
compression
branch
from
06976f6
to
f1af547
Compare
|
Discussed in t-infra meeting, decided to raise to 64 MB rather than 128. That should help avoid some of the impact for low-memory virtual machines (e.g., for free or cheap cloud hosting) while still giving us a majority of the benefits for most files, with only extended tarballs benefiting significantly from the larger compression window. |
pietroalbini
approved these changes
Feb 27, 2023
Mark-Simulacrum
added a commit
to Mark-Simulacrum/rust
that referenced
this pull request
Feb 27, 2023
This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for static.rust-lang.org, download times, and decompression time. This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup).
|
Has this been tested end-to-end? We're usually decompression bound, not network or disk IO, in rustup these days. Tuning for faster decompression is one of the tuits sitting around for rustup, and for instance, moving to async + tokio will drive up memory pressure rather than alleviate it. I'm obviously glad that we're worrying about user experience, but 15% decrease on network is not going to be a 15% decrease in 'time to install'. |
|
(Followed up in zulip) |
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this pull request
Feb 28, 2023
…troalbini Bump rust-installer This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for static.rust-lang.org, download times, and decompression time. This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup).
Mark-Simulacrum
added a commit
to Mark-Simulacrum/rust
that referenced
this pull request
Mar 7, 2023
This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for static.rust-lang.org, download times, and decompression time. This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup).
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 8, 2023
…oalbini Import rust-installer & adjust compression settings This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for static.rust-lang.org, download times, and decompression time. This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup). To simplify iteration on compression settings this also imports the rust-lang/rust-installer submodule, it is now hosted fully inside rust-lang/rust. Once we land this I'll file a followup to add a note to that repo and we can subsequently archive it. -- CI times for dist-x86_64-linux builds: * threads=6, master - 2h 50m * threads=1, new - 3h 40m * threads=6, new - 2h 50m
saethlin
pushed a commit
to saethlin/miri
that referenced
this pull request
Mar 11, 2023
Import rust-installer & adjust compression settings This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for static.rust-lang.org, download times, and decompression time. This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup). To simplify iteration on compression settings this also imports the rust-lang/rust-installer submodule, it is now hosted fully inside rust-lang/rust. Once we land this I'll file a followup to add a note to that repo and we can subsequently archive it. -- CI times for dist-x86_64-linux builds: * threads=6, master - 2h 50m * threads=1, new - 3h 40m * threads=6, new - 2h 50m
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adjusts our compression settings for xz compressed files, which should reduce download sizes by around 5-15%, at the cost of ~44% longer compression times. Given that compression happens almost universally in CI, rather than something humans are directly waiting for, this tradeoff feels worth it.
If we end up seeing this be too significant an increase in CI times, moving the more aggressive compression solely to promote-release should be relatively straightforward and give most of the benefits.
I expect we'll need to iterate a few times on the exact parameters as we try to run this through rust-lang/rust CI, but unfortunately I expect it'll need the full CI so I think merging this makes sense after review rather than trying to iterate through try builds.