Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 7/5/2022 10:32:06 PM #72030

Closed
performanceautofiler bot opened this issue Jul 12, 2022 · 5 comments
Closed

[Perf] Changes at 7/5/2022 10:32:06 PM #72030

performanceautofiler bot opened this issue Jul 12, 2022 · 5 comments
Assignees
@tannergooding
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jul 12, 2022 •
edited by DrewScoggins
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchF.MDLLoops

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 658.14 ms 695.60 ms 1.06 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchF.MDLLoops*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchF.MDLLoops.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 695.6042480769231 > 688.2603742326924.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/21/2022 4:54:11 AM, 6/7/2022 5:42:36 AM, 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -40.861647117168786 (T) = (0 -696232577.1907164) / Math.Sqrt((12804781397326.438 / (31)) + (24144080383239.016 / (45))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (45) - 2, .025) and -0.06066058823673496 = (656414111.0853835 - 696232577.1907164) / 656414111.0853835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchI.MDMulMatrix

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 644.48 ms 861.47 ms 1.34 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchI.MDMulMatrix*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDMulMatrix.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 861.4729290714286 > 676.9384600500001.
IsChangePoint: Marked as a change because one of 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -318.2377784562592 (T) = (0 -865401784.0575421) / Math.Sqrt((225348633110.33362 / (31)) + (20803085904355.203 / (44))) is less than -1.99299712588925 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (44) - 2, .025) and -0.3419061262248373 = (644904861.1859034 - 865401784.0575421) / 644904861.1859034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Jul 12, 2022
@DrewScoggins DrewScoggins removed the untriaged New issue has not been triaged by the area owner label Jul 12, 2022
@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues Jul 12, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 12, 2022
@DrewScoggins DrewScoggins added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jul 12, 2022
@DrewScoggins
Copy link
Member

Related to this commit: #70271

@danmoseley danmoseley added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jul 12, 2022
@ghost
Copy link

ghost commented Jul 12, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchF.MDLLoops

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 658.14 ms 695.60 ms 1.06 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchF.MDLLoops*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchF.MDLLoops.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 695.6042480769231 > 688.2603742326924.
IsChangePoint: Marked as a change because one of 5/11/2022 7:02:26 PM, 5/21/2022 4:54:11 AM, 6/7/2022 5:42:36 AM, 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -40.861647117168786 (T) = (0 -696232577.1907164) / Math.Sqrt((12804781397326.438 / (31)) + (24144080383239.016 / (45))) is less than -1.9925434951804422 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (45) - 2, .025) and -0.06066058823673496 = (656414111.0853835 - 696232577.1907164) / 656414111.0853835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 338db1a1ba498f308797ce7a54c0ed54e7286d50
Compare ffcb0149763663a55a79784be216afe81f6bf76c
Diff Diff

Regressions in Benchstone.MDBenchI.MDMulMatrix

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 644.48 ms 861.47 ms 1.34 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Benchstone.MDBenchI.MDMulMatrix*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDMulMatrix.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 861.4729290714286 > 676.9384600500001.
IsChangePoint: Marked as a change because one of 7/5/2022 7:01:28 PM, 7/12/2022 6:44:07 AM falls between 7/3/2022 4:26:07 AM and 7/12/2022 6:44:07 AM.
IsRegressionStdDev: Marked as regression because -318.2377784562592 (T) = (0 -865401784.0575421) / Math.Sqrt((225348633110.33362 / (31)) + (20803085904355.203 / (44))) is less than -1.99299712588925 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (44) - 2, .025) and -0.3419061262248373 = (644904861.1859034 - 865401784.0575421) / 644904861.1859034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: tannergooding
Labels:

tenet-performance, tenet-performance-benchmarks, area-CodeGen-coreclr, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, CoreClr, x64
Milestone: -

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Jul 12, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 7.0.0 milestone Jul 12, 2022
@tannergooding tannergooding modified the milestones: 7.0.0, 8.0.0 Aug 1, 2022
@mrsharm
Copy link
Member

mrsharm commented Aug 8, 2022

This regression showed up in our perf report for preview 7. The details are as follows:

Benchstone.MDBenchI.MDMulMatrix.Test

Result Ratio Alloc Delta Operating System Bit Processor Name
Faster 1.66 -288 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 1.71 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 1.23 +672 macOS Monterey 12.3 Arm64 Apple M1 Max
Slower 0.76 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.75 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.80 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.79 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.76 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Slower 0.81 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.86 +0 Windows 11 X64 AMD Ryzen 9 3950X
Same 1.04 +0 Windows 11 X64 AMD Ryzen 9 5900X
Same 0.99 +0 Windows 11 X64 AMD Ryzen 9 5950X
Slower 0.81 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.77 +0 Windows 11 X64 Intel Core i9-10900K CPU 3.70GHz
Slower 0.61 +0 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Slower 0.74 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.86 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.74 +0 ubuntu 18.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.97 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.72 +0 ubuntu 20.04 X64 Intel Core i9-10900K CPU 3.70GHz
Same 0.99 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 1.03 +0 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 0.98 +0 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.73 +0 macOS Big Sur 11.6.8 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 0.73 +0 macOS Monterey 12.3.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Slower 0.72 +672 macOS Monterey 12.4 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

@mrsharm mrsharm mentioned this issue Aug 12, 2022
Closed
24 tasks
@BruceForstall
Copy link
Member

This regression is expected, as per discussion in #70271. Follow-up work items have been filed to improve this.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed ubuntu 18.04 labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
7 participants
@jeffhandley @danmoseley @DrewScoggins @tannergooding @BruceForstall @JulieLeeMSFT @mrsharm