perf: faster prefix record deser #777
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nichmor
commented
Jul 12, 2024
nichmor
changed the title
baszalmstra
reviewed
Jul 12, 2024
| @@ -23,6 +23,7 @@ purl = { workspace = true, features = ["serde"] } | |||
| rattler_digest = { path = "../rattler_digest", version = "0.19.4", default-features = false, features = ["serde"] } | |||
| rattler_macros = { path = "../rattler_macros", version = "0.19.4", default-features = false } | |||
| regex = { workspace = true } | |||
| simd-json = { version = "*", features = ["serde_impl"]} | |||
There was a problem hiding this comment.
Can you add a minimal version here?
There was a problem hiding this comment.
made this workspace=true and added simd-json in root Cargo.toml
|
I pushed a commit that removes the unsafe usage: 11c8600 |
baszalmstra
approved these changes
Jul 12, 2024
Merged
baszalmstra
added a commit
that referenced
this pull request
Jul 15, 2024
## 🤖 New release * `file_url`: 0.1.2 -> 0.1.3 * `rattler`: 0.26.5 -> 0.27.0 * `rattler_cache`: 0.1.1 -> 0.1.2 * `rattler_conda_types`: 0.26.0 -> 0.26.1 * `rattler_digest`: 0.19.4 -> 0.19.5 * `rattler_package_streaming`: 0.21.4 -> 0.21.5 * `rattler_networking`: 0.20.9 -> 0.20.10 * `rattler_shell`: 0.21.0 -> 0.21.1 * `rattler_lock`: 0.22.13 -> 0.22.14 * `rattler_repodata_gateway`: 0.21.0 -> 0.21.1 * `rattler_solve`: 0.25.0 -> 0.25.1 * `rattler_libsolv_c`: 0.19.4 -> 0.19.5 * `rattler_virtual_packages`: 0.19.16 -> 0.19.17 * `rattler_index`: 0.19.18 -> 0.19.19 <details><summary><i><b>Changelog</b></i></summary><p> ## `file_url` <blockquote> ## [0.1.3](file_url-v0.1.2...file_url-v0.1.3) - 2024-07-15 ### Other - update Cargo.toml dependencies </blockquote> ## `rattler` <blockquote> ## [0.27.0](rattler-v0.26.5...rattler-v0.27.0) - 2024-07-15 ### Fixed - unclobber issue when packages are named differently ([#776](#776)) ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_cache` <blockquote> ## [0.1.2](rattler_cache-v0.1.1...rattler_cache-v0.1.2) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_conda_types` <blockquote> ## [0.26.1](rattler_conda_types-v0.26.0...rattler_conda_types-v0.26.1) - 2024-07-15 ### Other - PrefixRecord deserialization using simd ([#777](#777)) </blockquote> ## `rattler_digest` <blockquote> ## [0.19.5](rattler_digest-v0.19.4...rattler_digest-v0.19.5) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_package_streaming` <blockquote> ## [0.21.5](rattler_package_streaming-v0.21.4...rattler_package_streaming-v0.21.5) - 2024-07-15 ### Other - bump zip to 2.1.3 ([#772](#772)) </blockquote> ## `rattler_networking` <blockquote> ## [0.20.10](rattler_networking-v0.20.9...rattler_networking-v0.20.10) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_shell` <blockquote> ## [0.21.1](rattler_shell-v0.21.0...rattler_shell-v0.21.1) - 2024-07-15 ### Other - update Cargo.toml dependencies </blockquote> ## `rattler_lock` <blockquote> ## [0.22.14](rattler_lock-v0.22.13...rattler_lock-v0.22.14) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_repodata_gateway` <blockquote> ## [0.21.1](rattler_repodata_gateway-v0.21.0...rattler_repodata_gateway-v0.21.1) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_solve` <blockquote> ## [0.25.1](rattler_solve-v0.25.0...rattler_solve-v0.25.1) - 2024-07-15 ### Other - update Cargo.toml dependencies </blockquote> ## `rattler_libsolv_c` <blockquote> ## [0.19.5](rattler_libsolv_c-v0.19.4...rattler_libsolv_c-v0.19.5) - 2024-07-15 ### Other - bump dependencies and remove unused ones ([#771](#771)) </blockquote> ## `rattler_virtual_packages` <blockquote> ## [0.19.17](rattler_virtual_packages-v0.19.16...rattler_virtual_packages-v0.19.17) - 2024-07-15 ### Other - updated the following local packages: rattler_conda_types </blockquote> ## `rattler_index` <blockquote> ## [0.19.19](rattler_index-v0.19.18...rattler_index-v0.19.19) - 2024-07-15 ### Other - updated the following local packages: rattler_conda_types, rattler_digest, rattler_package_streaming </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
based on this blog https://blog.datalust.co/deserializing-json-really-fast/ I've tried to choose simd_json for our deserialization to PrefixRecord as it seems that it has a lot of samples in the flamegraph
In the benchmarks, having 42 JSON files in the conda_meta folder and using simd_json will give us the following results:
using old way:
We can see an increase of
+5.4835%