Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the unstable option to reduce the binary size of dynamic library… #118636

Merged
merged 1 commit into from
Jan 27, 2024

Conversation

h1467792822
Copy link
Contributor

@h1467792822 h1467792822 commented Dec 5, 2023

Motivation

The average length of symbol names in the rust standard library is about 100 bytes, while the average length of symbol names in the C++ standard library is about 65 bytes. In some embedded environments where dynamic library are widely used, rust dynamic library symbol name space hash become one of the key bottlenecks of application, Especially when the existing C/C++ module is reconstructed into the rust module.

The unstable option -Z symbol_mangling_version=hashed is added to solve the bottleneck caused by too long dynamic library symbol names.

Test data

The following is a set of test data on the ubuntu 18.04 LTS environment. With this plug-in, the space saving rate of dynamic libraries can reach about 20%.

The test object is the standard library of rust (built based on Xargo), tokio crate, and hyper crate.

The contents of the Cargo.toml file in the construction project of the three dynamic libraries are as follows:

# Cargo.toml
[profile.release]
panic = "abort"
opt-leve="z"
codegen-units=1
strip=true
debug=true

The built dynamic library also removes the .rustc segments that are not needed at run time and then compares the size. The detailed data is as follows:

  1. libstd.so
symbol_mangling_version size saving rate
legacy 804896
hashed 608288 0.244
v0 858144
hashed 608288 0.291
  1. libhyper.so
symbol_mangling_version(libhyper.so) symbol_mangling_version(libstd.so) size saving rate
legacy legacy 866312
hashed legacy 645128 0.255
legacy hashed 854024
hashed hashed 632840 0.259

@rustbot
Copy link
Collaborator

rustbot commented Dec 5, 2023

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @cjgillot (or someone else) soon.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

  • @rustbot author: the review is finished, PR author should check the comments and take action accordingly
  • @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 5, 2023
@rust-log-analyzer

This comment has been minimized.

@h1467792822
Copy link
Contributor Author

r? @m-ou-se

@rustbot rustbot assigned m-ou-se and unassigned cjgillot Dec 5, 2023
@m-ou-se
Copy link
Member

m-ou-se commented Dec 14, 2023

20%

Exciting results!

This will have to be reviewed by the compiler team. I imagine they might want an MCP first.

r? compiler

@rustbot rustbot assigned compiler-errors and unassigned m-ou-se Dec 14, 2023
@compiler-errors
Copy link
Member

Yes, please open an MCP. Also this needs a test or something that demonstrates it's working correctly.

@rustbot author

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 14, 2023
@bors
Copy link
Contributor

bors commented Dec 14, 2023

☔ The latest upstream changes (presumably #118937) made this pull request unmergeable. Please resolve the merge conflicts.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@michaelwoerister
Copy link
Member

Thanks for the PR and the MCP, @h1467792822! Looks very promising 🙂

r? @michaelwoerister

@h1467792822
Copy link
Contributor Author

@rustbot ready

bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 25, 2024
Add the unstable option  to reduce the binary size of dynamic library…

# Motivation

The average length of symbol names in the rust standard library is about 100 bytes, while the average length of symbol names in the C++ standard library is about 65 bytes. In some embedded environments where dynamic library are widely used, rust dynamic library symbol name space hash become one of the key bottlenecks of application, Especially when the existing C/C++ module is reconstructed into the rust module.

The unstable option `-Z symbol_mangling_version=hashed` is added to solve the bottleneck caused by too long dynamic library symbol names.

## Test data

The following is a set of test data on the ubuntu 18.04 LTS environment. With this plug-in, the space saving rate of dynamic libraries can reach about 20%.

The test object is the standard library of rust (built based on Xargo), tokio crate, and hyper crate.

The contents of the Cargo.toml file in the construction project of the three dynamic libraries are as follows:

```txt
# Cargo.toml
[profile.release]
panic = "abort"
opt-leve="z"
codegen-units=1
strip=true
debug=true
```
The built dynamic library also removes the `.rustc` segments that are not needed at run time and then compares the size. The detailed data is as follows:

1. libstd.so
> | symbol_mangling_version | size | saving rate |
> | --- | --- | --- |
> | legacy | 804896 ||
> | hashed | 608288 | 0.244 |
> | v0 | 858144 ||
> | hashed | 608288 | 0.291 |

2. libhyper.so
> | symbol_mangling_version(libhyper.so) | symbol_mangling_version(libstd.so) | size | saving rate |
> | --- | --- | --- | --- |
> | legacy | legacy | 866312 ||
> | hashed | legacy | 645128 |0.255|
> | legacy | hashed | 854024 ||
> | hashed | hashed | 632840 |0.259|
@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Contributor

bors commented Jan 25, 2024

💔 Test failed - checks-actions

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 25, 2024
…ed -Z unstable-options` to shorten symbol names by replacing them with a digest.

Enrich test cases
@h1467792822
Copy link
Contributor Author

@michaelwoerister Sorry, the code I submitted failed the test again.
I compared the other test cases and found that I still need to add # ignore-cross-compile to the Makefile (I mistakenly deleted it last time I submitted it).
I verify in the local musl environment. After the modification, no error is reported when ./x.py test tests/run-make/symbol-mangling-hashed --stage 1 --target x86_64-unknown-linux-musl is executed.
If the modification is incorrect, please give me some advice again. Thank you!

@michaelwoerister
Copy link
Member

Thanks, @h1467792822 -- that seems like an appropriate solution. The cross-compiling issue is not related to what's being tested here, as far as I can tell.

@bors r+

@bors
Copy link
Contributor

bors commented Jan 26, 2024

📌 Commit 6e53e66 has been approved by michaelwoerister

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 26, 2024
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 26, 2024
Add the unstable option  to reduce the binary size of dynamic library…

# Motivation

The average length of symbol names in the rust standard library is about 100 bytes, while the average length of symbol names in the C++ standard library is about 65 bytes. In some embedded environments where dynamic library are widely used, rust dynamic library symbol name space hash become one of the key bottlenecks of application, Especially when the existing C/C++ module is reconstructed into the rust module.

The unstable option `-Z symbol_mangling_version=hashed` is added to solve the bottleneck caused by too long dynamic library symbol names.

## Test data

The following is a set of test data on the ubuntu 18.04 LTS environment. With this plug-in, the space saving rate of dynamic libraries can reach about 20%.

The test object is the standard library of rust (built based on Xargo), tokio crate, and hyper crate.

The contents of the Cargo.toml file in the construction project of the three dynamic libraries are as follows:

```txt
# Cargo.toml
[profile.release]
panic = "abort"
opt-leve="z"
codegen-units=1
strip=true
debug=true
```
The built dynamic library also removes the `.rustc` segments that are not needed at run time and then compares the size. The detailed data is as follows:

1. libstd.so
> | symbol_mangling_version | size | saving rate |
> | --- | --- | --- |
> | legacy | 804896 ||
> | hashed | 608288 | 0.244 |
> | v0 | 858144 ||
> | hashed | 608288 | 0.291 |

2. libhyper.so
> | symbol_mangling_version(libhyper.so) | symbol_mangling_version(libstd.so) | size | saving rate |
> | --- | --- | --- | --- |
> | legacy | legacy | 866312 ||
> | hashed | legacy | 645128 |0.255|
> | legacy | hashed | 854024 ||
> | hashed | hashed | 632840 |0.259|
@bors
Copy link
Contributor

bors commented Jan 26, 2024

⌛ Testing commit 6e53e66 with merge eb8e55a...

@bors
Copy link
Contributor

bors commented Jan 26, 2024

💥 Test timed out

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 26, 2024
@fmease
Copy link
Member

fmease commented Jan 26, 2024

@bors retry timeout (reoccurring)

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 26, 2024
@rust-log-analyzer
Copy link
Collaborator

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

@bors
Copy link
Contributor

bors commented Jan 27, 2024

⌛ Testing commit 6e53e66 with merge 04521fd...

Nadrieril added a commit to Nadrieril/rust that referenced this pull request Jan 27, 2024
Add the unstable option  to reduce the binary size of dynamic library…

# Motivation

The average length of symbol names in the rust standard library is about 100 bytes, while the average length of symbol names in the C++ standard library is about 65 bytes. In some embedded environments where dynamic library are widely used, rust dynamic library symbol name space hash become one of the key bottlenecks of application, Especially when the existing C/C++ module is reconstructed into the rust module.

The unstable option `-Z symbol_mangling_version=hashed` is added to solve the bottleneck caused by too long dynamic library symbol names.

## Test data

The following is a set of test data on the ubuntu 18.04 LTS environment. With this plug-in, the space saving rate of dynamic libraries can reach about 20%.

The test object is the standard library of rust (built based on Xargo), tokio crate, and hyper crate.

The contents of the Cargo.toml file in the construction project of the three dynamic libraries are as follows:

```txt
# Cargo.toml
[profile.release]
panic = "abort"
opt-leve="z"
codegen-units=1
strip=true
debug=true
```
The built dynamic library also removes the `.rustc` segments that are not needed at run time and then compares the size. The detailed data is as follows:

1. libstd.so
> | symbol_mangling_version | size | saving rate |
> | --- | --- | --- |
> | legacy | 804896 ||
> | hashed | 608288 | 0.244 |
> | v0 | 858144 ||
> | hashed | 608288 | 0.291 |

2. libhyper.so
> | symbol_mangling_version(libhyper.so) | symbol_mangling_version(libstd.so) | size | saving rate |
> | --- | --- | --- | --- |
> | legacy | legacy | 866312 ||
> | hashed | legacy | 645128 |0.255|
> | legacy | hashed | 854024 ||
> | hashed | hashed | 632840 |0.259|
@bors
Copy link
Contributor

bors commented Jan 27, 2024

☀️ Test successful - checks-actions
Approved by: michaelwoerister
Pushing 04521fd to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 27, 2024
@bors bors merged commit 04521fd into rust-lang:master Jan 27, 2024
12 checks passed
@rustbot rustbot added this to the 1.77.0 milestone Jan 27, 2024
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (04521fd): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
2.7% [2.7%, 2.7%] 1
Improvements ✅
(primary)
-3.2% [-3.2%, -3.2%] 1
Improvements ✅
(secondary)
-3.7% [-3.7%, -3.7%] 1
All ❌✅ (primary) -3.2% [-3.2%, -3.2%] 1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 660.976s -> 660.806s (-0.03%)
Artifact size: 308.12 MiB -> 308.11 MiB (-0.00%)

@h1467792822 h1467792822 deleted the dev branch January 30, 2024 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
has-merge-commits PR has merge commits, merge with caution. merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants