Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build with PGO on x86_64 ubuntu and windows #678

Merged
merged 4 commits into from
Jul 3, 2023
Merged

build with PGO on x86_64 ubuntu and windows #678

merged 4 commits into from
Jul 3, 2023

Conversation

davidhewitt
Copy link
Contributor

@davidhewitt davidhewitt commented Jun 20, 2023

This is a brief experiment to see what happens if we use the test suite to gather profiling data, to feed back into the build for PGO.

If benchmarks look promising, this branch needs extending to also build the uploaded wheels using PGO in the same way.

Selected Reviewer: @adriangb

@codecov-commenter
Copy link

codecov-commenter commented Jun 20, 2023

Codecov Report

Merging #678 (383f497) into main (d285975) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #678   +/-   ##
=======================================
  Coverage   93.63%   93.63%           
=======================================
  Files          99       99           
  Lines       13913    13913           
  Branches       25       25           
=======================================
  Hits        13028    13028           
  Misses        879      879           
  Partials        6        6           

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d285975...383f497. Read the comment docs.

@davidhewitt davidhewitt force-pushed the dh/pgo branch 3 times, most recently from 2db5fa0 to 39cd650 Compare June 21, 2023 08:25
@codspeed-hq
Copy link

codspeed-hq bot commented Jun 21, 2023

CodSpeed Performance Report

Merging #678 will improve performances by 37.33%

Comparing dh/pgo (383f497) with main (d285975)

Summary

🔥 39 improvements
✅ 87 untouched benchmarks

Benchmarks breakdown

Benchmark main dh/pgo Change
🔥 test_complete_core_lax 1.4 ms 1.2 ms +17.51%
🔥 test_complete_core_strict 1.4 ms 1.2 ms +17.94%
🔥 test_build_schema 4.1 ms 3.5 ms +16.95%
🔥 test_core_python_fs 353.8 µs 294.5 µs +20.16%
🔥 test_core_json_fs 859.2 µs 771 µs +11.44%
🔥 test_core_python 509.4 µs 452 µs +12.7%
🔥 test_definition_model_core 2.1 ms 1.8 ms +13.59%
🔥 test_list_of_dict_models_core 521.8 µs 451.1 µs +15.68%
🔥 test_list_of_ints_core_py 2.6 ms 2.2 ms +18.03%
🔥 test_set_of_ints_core 3.8 ms 3.2 ms +17.31%
🔥 test_set_of_ints_core_duplicates 2.4 ms 1.8 ms +33.45%
🔥 test_set_of_ints_core_length 3.9 ms 3.3 ms +17.15%
🔥 test_set_of_ints_core_json_duplicates 5.4 ms 4.7 ms +13.67%
🔥 test_frozenset_of_ints_core 1.3 ms 1.1 ms +16.8%
🔥 test_frozenset_of_ints_duplicates_core 999.9 µs 807.9 µs +23.77%
🔥 test_dict_of_ints_core 5.9 ms 4.7 ms +24.99%
🔥 test_dict_of_any_core 4.2 ms 3.4 ms +25.72%
🔥 test_dict_of_ints_core_json 15.6 ms 13.4 ms +16.66%
🔥 test_many_models_core_dict 5.5 ms 5 ms +11.4%
🔥 test_many_models_core_model 14.7 ms 12.9 ms +13.97%
🔥 test_list_of_nullable_core 642.9 µs 540 µs +19.05%
🔥 test_tuple_many_variable 31.5 µs 28.4 µs +11.13%
🔥 test_arguments 44.2 µs 38.4 µs +15.02%
🔥 test_definition_in_tree 6.1 ms 4.6 ms +31.69%
🔥 test_definition_out_of_tree 9.9 ms 7.4 ms +33.06%
🔥 test_model_instance 51.6 µs 46 µs +12.24%
🔥 test_core_root_model 112 µs 98.1 µs +14.09%
🔥 test_tagged_union_int_keys_python 33.6 µs 30.4 µs +10.45%
🔥 test_core_dict 286.6 µs 246.9 µs +16.08%
🔥 test_core_dict_filter 296.4 µs 256.9 µs +15.37%
🔥 test_core_json 395.1 µs 338.5 µs +16.71%
🔥 test_json_direct_list_str 926.7 µs 712.1 µs +30.15%
🔥 test_python_json_list_str 709.9 µs 580 µs +22.4%
🔥 test_json_any_list_str 1.6 ms 1.3 ms +27.06%
🔥 test_json_direct_list_int 905.2 µs 659.2 µs +37.33%
🔥 test_json_any_list_int 1.6 ms 1.3 ms +22.48%
🔥 test_python_json_list_int 706.3 µs 576.4 µs +22.55%
🔥 test_python_json_list_none 685.6 µs 555.9 µs +23.33%
🔥 test_model_list_core_json 968.3 µs 726.1 µs +33.35%

Copy link
Member

@adriangb adriangb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just based on the benchmark improvements this seems like a no brainer to me

.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
name: pypi_files
path: dist

build-pgo:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do two builds?

Copy link
Contributor Author

@davidhewitt davidhewitt Jun 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problems I have been encountering with the merged step:

  • (very hard in a merged step) we need to pass the correct PGO data per interpreter build as a rust flag. I think maturin/maturin-action currently assumes flags are same for all interpreters.
  • (fixable in a merged step) we need to run pytest (or other PGO sampling) with the correct interpreter for the build.

I have started with this split approach to verify the end-to-end approach works, and limited just to what I assume is the most commonly used platform. It would be good to merge back into a single step and expand to more OS versions. Think I need upstream support in maturin to achieve that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense.

.github/workflows/codspeed.yml Outdated Show resolved Hide resolved
.github/workflows/codspeed.yml Show resolved Hide resolved
@davidhewitt
Copy link
Contributor Author

This is now tidied up and uses tests/benchmarks for the PGO input. I believe it is good to merge, unless we want to also apply PGO builds to other major OS / arch versions.

@davidhewitt davidhewitt force-pushed the dh/pgo branch 2 times, most recently from 84efead to 467e6d7 Compare July 3, 2023 13:31
@davidhewitt davidhewitt force-pushed the dh/pgo branch 5 times, most recently from c09a25e to 391840b Compare July 3, 2023 14:03
@davidhewitt davidhewitt force-pushed the dh/pgo branch 2 times, most recently from ccd4017 to af731d5 Compare July 3, 2023 14:16
@davidhewitt davidhewitt changed the title try pgo for benchmarks run build with PGO on x86_64 ubuntu and windows Jul 3, 2023
@davidhewitt davidhewitt force-pushed the dh/pgo branch 2 times, most recently from e40c472 to 383f497 Compare July 3, 2023 14:44
@davidhewitt
Copy link
Contributor Author

please review

@samuelcolvin
Copy link
Member

would be great in future if we can have a make command to do this, e.g. for people running benchmarks on their M1 macbooks.

@samuelcolvin samuelcolvin merged commit 9ad1425 into main Jul 3, 2023
60 checks passed
@samuelcolvin samuelcolvin deleted the dh/pgo branch July 3, 2023 16:23
@davidhewitt
Copy link
Contributor Author

I will open an upstream issue on maturin-action, maybe it makes sense to support there (or directly in maturin).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Full Build cause CI to do a full build ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants