Run crossgen in parallel in crossgen_comparison.py #33175
Conversation
jashook
force-pushed
the
cross-crossgen-parallel
branch
from
3c9c5c3
to
0752950
Compare
|
I believe this is finally ready for review @echesakovMSFT @dotnet/runtime-infrastructure ptal |
|
Will post runtime improvements once the run finishes |
| @@ -38,7 +38,7 @@ jobs: | |||
| archType: arm | |||
| platform: Linux_arm | |||
| container: | |||
| image: ubuntu-16.04-cross-14.04-23cacb0-20200121150126 | |||
| image: ubuntu-18.04-cross-arm-16.04-09ec757-20200324125113 | |||
There was a problem hiding this comment.
Why did we need to upgrade the image here?
There was a problem hiding this comment.
The script needs python3.6+
There was a problem hiding this comment.
Its also generally not needed for us to target 14.04 anymore as 14.04 is eol
|
|
||
| # Create baseline output on the host (x64) machine | ||
| - task: PythonScript@0 | ||
| displayName: Create cross-platform crossgen baseline | ||
| inputs: | ||
| scriptSource: 'filePath' | ||
| scriptPath: $(coreClrRepoRoot)/tests/scripts/crossgen_comparison.py | ||
| pythonInterpreter: /usr/bin/python3 |
There was a problem hiding this comment.
Do we need to add python3 to our linux requirements markdown file?
There was a problem hiding this comment.
We should be one python3, seeing that python2 is eol. We have not exactly mandated it yet.
Time in seconds
|
|
This will leave our cross-crossgen job with ~3 minutes of helix setup time (download, zip and docker) and ~2 minutes of execution time. Total time is reduced ~40% with this change. |
|
Very nice. That matches the ratio between Crossgen / Crossgen2 framework compilation times in "build-test generatelayoutonly" (where there's a straight linear loop over the assemblies) and SuperIlc (which uses parallelization much like you do, just written in C#), something like 3:1, almost reaching the 4:1 ideal speedup on my 4-core box. |
ghost
locked as resolved and limited conversation to collaborators
This will hopefully reduce the runtime of crossgen_compairision.py and address #33172