Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use ZIP file size metadata to allocate string #13032

Merged
merged 2 commits into from
Aug 21, 2024

Conversation

MichaReiser
Copy link
Member

@MichaReiser MichaReiser commented Aug 21, 2024

Summary

We spend about 1.4% in VendoredFileSystem::read_to_string. Let's pre-allocate the string with the right size by using the zip file metadata
as a heuristic (don't trust it!)

Test Plan

cargo test

I verified that total spend in VendoredFileSystem::read_to_string goes down to 1.1% and it no longer shows up as a large contributor to malloc

@AlexWaygood
Copy link
Member

Hmm. In #11863 (comment), you argued against doing this

Copy link

codspeed-hq bot commented Aug 21, 2024

CodSpeed Performance Report

Merging #13032 will improve performances by 8.11%

Comparing vendored-alloc-string-with-right-size (170ffb4) with main (a35cdbb)

Summary

⚡ 1 improvements
✅ 31 untouched benchmarks

Benchmarks breakdown

Benchmark main vendored-alloc-string-with-right-size Change
linter/default-rules[numpy/globals.py] 183.9 µs 170.1 µs +8.11%

@MichaReiser
Copy link
Member Author

Yeah, I now saw in the benchmarks (still need to verify that it helped) that I was wrong.

Reading through the code:

    fn read_to_string(&mut self, buf: &mut String) -> Result<usize> {
        default_read_to_string(self, buf, None)
    }

the last argument is the size hint... Oh well, it is None

Copy link
Contributor

github-actions bot commented Aug 21, 2024

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining, sounds good!

(Is it worth adding a comment explaining the reasoning you just gave me?)

@MichaReiser MichaReiser added the red-knot Multi-file analysis & type inference label Aug 21, 2024
@MichaReiser MichaReiser force-pushed the vendored-alloc-string-with-right-size branch from 2d42a4b to 170ffb4 Compare August 21, 2024 12:44
@MichaReiser MichaReiser enabled auto-merge (squash) August 21, 2024 12:46
@MichaReiser MichaReiser merged commit 785c399 into main Aug 21, 2024
18 checks passed
@MichaReiser MichaReiser deleted the vendored-alloc-string-with-right-size branch August 21, 2024 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
red-knot Multi-file analysis & type inference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants