Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191

Closed
Tracked by #79004
adamsitnik opened this issue Feb 11, 2022 · 9 comments
Closed
Tracked by #79004

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191

adamsitnik opened this issue Feb 11, 2022 · 9 comments
Assignees
@tannergooding
Labels
area-System.Numerics tenet-performance Performance related issue
Milestone
8.0.0

Comments

@adamsitnik
Copy link
Member

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed, it might be related to #59415

image

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

It seems that @performanceautofiler has missed the regression (cc @DrewScoggins ): https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark.html

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

Result Base Diff Ratio Modality Operating System Bit Processor Name
Slower 4.67 8.10 0.58 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 3.72 7.00 0.53 Windows 11 X64 AMD Ryzen 9 5900X
Same 5.56 7.91 0.70 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 7.99 11.94 0.67 several? Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Same 5.62 7.03 0.80 Windows 10 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 5.04 6.85 0.74 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 5.20 8.06 0.65 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 7.80 10.54 0.74 Windows 11 X64 Unknown processor
Same 9.87 9.32 1.06 multimodal Windows 11 X64 Unknown processor
Slower 4.22 7.39 0.57 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 5.40 10.62 0.51 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 9.77 12.18 0.80 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Slower 7.62 11.41 0.67 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 6.06 8.61 0.70 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 5.99 8.41 0.71 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 6.04 8.30 0.73 ubuntu 20.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 10.94 11.56 0.95 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Slower 7.16 11.15 0.64 bimodal Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 34.35 31.63 1.09 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 7.07 11.71 0.60 macOS Big Sur 11.6.3 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 6.35 10.57 0.60 macOS Big Sur 11.4 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)

cc @tannergooding
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Feb 11, 2022
@ghost
Copy link

ghost commented Feb 11, 2022

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed, it might be related to #59415

image

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

It seems that @performanceautofiler has missed the regression (cc @DrewScoggins ): https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark.html

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

Result Base Diff Ratio Modality Operating System Bit Processor Name
Slower 4.67 8.10 0.58 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 3.72 7.00 0.53 Windows 11 X64 AMD Ryzen 9 5900X
Same 5.56 7.91 0.70 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 7.99 11.94 0.67 several? Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell)
Same 5.62 7.03 0.80 Windows 10 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 5.04 6.85 0.74 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 5.20 8.06 0.65 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 7.80 10.54 0.74 Windows 11 X64 Unknown processor
Same 9.87 9.32 1.06 multimodal Windows 11 X64 Unknown processor
Slower 4.22 7.39 0.57 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 5.40 10.62 0.51 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 9.77 12.18 0.80 centos 7 X64 Intel Xeon CPU E5530 2.40GHz
Slower 7.62 11.41 0.67 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 6.06 8.61 0.70 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 5.99 8.41 0.71 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 6.04 8.30 0.73 ubuntu 20.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same 10.94 11.56 0.95 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Slower 7.16 11.15 0.64 bimodal Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 34.35 31.63 1.09 Windows 10 Arm Microsoft SQ1 3.0 GHz
Slower 7.07 11.71 0.60 macOS Big Sur 11.6.3 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 6.35 10.57 0.60 macOS Big Sur 11.4 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)

cc @tannergooding

Author: adamsitnik
Assignees: -
Labels:

area-System.Numerics, tenet-performance
Milestone: -

@adamsitnik
Copy link
Member Author

Same goes for System.Numerics.Tests.Perf_Matrix4x4.Transpose

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fSystem.Numerics.Tests.Perf_Matrix4x4.Transpose.html

image

Result Base Diff Ratio Alloc Delta Modality Operating System Bit Processor Name Base V Diff V
Slower 4.78 6.93 0.69 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores 6.0.121.56705 7.0.22.7608
Slower 3.73 6.99 0.53 +0 Windows 11 X64 AMD Ryzen 9 5900X 6.0.121.56705 7.0.22.7608
Slower 5.36 8.72 0.61 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz 6.0.121.56705 7.0.22.7608
Slower 8.56 11.67 0.73 +0 Windows 11 X64 Intel Core i5-4300U CPU 1.90GHz (Haswell) 6.0.121.56705 7.0.22.7608
Same 5.76 7.27 0.79 +0 Windows 10 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) 6.0.121.56705 7.0.22.7608
Same 5.31 6.99 0.76 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) 6.0.121.56705 7.0.22.7608
Slower 5.32 7.61 0.70 +0 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz 6.0.121.56705 7.0.22.7608
Slower 7.89 10.23 0.77 +0 Windows 11 X64 Unknown processor 6.0.121.56705 7.0.22.7608
Same 7.19 9.15 0.78 +0 Windows 11 X64 Unknown processor 6.0.121.56705 7.0.22.7608
Slower 4.68 7.52 0.62 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X 6.0.121.56705 7.0.22.7608
Slower 5.86 10.62 0.55 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz 6.0.121.56705 7.0.22.7608
Slower 10.83 13.40 0.81 +0 centos 7 X64 Intel Xeon CPU E5530 2.40GHz 6.0.121.56705 7.0.22.7608
Same 8.18 7.92 1.03 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge) 6.0.121.56705 7.0.22.7608
Slower 5.90 8.69 0.68 +0 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) 6.0.121.56705 7.0.22.7608
Slower 5.88 8.02 0.73 +0 ubuntu 18.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) 6.0.121.56705 7.0.22.7608
Slower 5.93 9.20 0.64 +0 bimodal ubuntu 20.04 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) 6.0.121.56705 7.0.22.7608
Same 12.77 12.40 1.03 +0 Windows 10 Arm64 Microsoft SQ1 3.0 GHz 6.0.121.56705 7.0.22.7608
Slower 6.47 10.07 0.64 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz 6.0.121.56705 7.0.22.7608
Same 30.33 33.24 0.91 +0 Windows 10 Arm Microsoft SQ1 3.0 GHz 6.0.121.56705 7.0.22.7608
Slower 7.54 11.13 0.68 +0 macOS Big Sur 11.6.3 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell) 6.0.121.56705 7.0.22.7608
Slower 6.42 12.87 0.50 +0 macOS Big Sur 11.4 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell) 6.0.121.56705 7.0.22.7608

@adamsitnik
Copy link
Member Author

And System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark:

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_Windows%2010.0.18362%2fSystem.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark.html

image

@adamsitnik
Copy link
Member Author

There are more, I won't list them all here.

@tannergooding
Copy link
Member

I'll have to look at the codegen for these. They haven't been changed in a long time and so I expect this is something that's changed in CSE or similar.

Notably, they could/should likely be simplified to use the xplat helper intrinsics now, rather than individual AdvSimd/Sse code paths

@adamsitnik adamsitnik mentioned this issue Feb 11, 2022
Closed
13 tasks
@DrewScoggins DrewScoggins mentioned this issue Feb 11, 2022
Open
@tannergooding tannergooding removed the untriaged New issue has not been triaged by the area owner label Mar 14, 2022
@tannergooding tannergooding self-assigned this Mar 14, 2022
@tannergooding tannergooding added this to the 7.0.0 milestone Mar 14, 2022
@dakersnar dakersnar modified the milestones: 7.0.0, 8.0.0 Aug 8, 2022
@jozkee
Copy link
Member

jozkee commented Oct 14, 2022 •
edited
Loading

Some benchmarks discussed here appeared in the 6.0 vs 7.0-rc2 perf report. Are they still under investigation? The historical data only goes back until october 2021 so we are unable to determine if this is noise nor to see what might've caused it.

  • System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark
  • System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark
  • System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark
  • System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark
  • System.Numerics.Tests.Perf_Matrix4x4.Transpose

System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Same 1.06 +0 ubuntu 18.04 Arm64 Unknown processor
Noise - +0 Windows 11 Arm64 Unknown processor
Same 1.05 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.03 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 0.98 +0 macOS Monterey 12.6 Arm64 Apple M1
Same 1.00 +0 macOS Monterey 12.6 Arm64 Apple M1 Max
Slower 0.70 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.59 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.50 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.46 +0 Windows 11 X64 AMD Ryzen 9 7950X
Slower 0.62 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 1.84 +0 debian 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.59 +0 ubuntu 18.04 X64 AMD Ryzen 9 5900X
Slower 0.25 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.55 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.77 +0 ubuntu 20.04 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Noise - +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.67 +0 macOS Big Sur 11.7 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 0.65 +0 macOS Monterey 12.6 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Faster 1.15 +0 ubuntu 18.04 Arm64 Unknown processor
Slower 0.67 +0 Windows 11 Arm64 Unknown processor
Faster 1.23 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 1.17 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Noise - +0 macOS Monterey 12.6 Arm64 Apple M1
Noise - +0 macOS Monterey 12.6 Arm64 Apple M1 Max
Same 0.96 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.73 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.63 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.67 +0 Windows 11 X64 AMD Ryzen 9 7950X
Same 0.91 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake) several?
Same 0.92 +0 debian 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.29 +0 ubuntu 18.04 X64 AMD Ryzen 9 5900X
Slower 0.31 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.67 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Same 0.92 +0 ubuntu 20.04 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Same 1.06 +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.92 +0 macOS Big Sur 11.7 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Noise - +0 macOS Monterey 12.6 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Same 1.05 +0 ubuntu 18.04 Arm64 Unknown processor
Same 0.91 +0 Windows 11 Arm64 Unknown processor
Same 1.04 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 1.59 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 0.89 +0 macOS Monterey 12.6 Arm64 Apple M1
Same 0.99 +0 macOS Monterey 12.6 Arm64 Apple M1 Max
Noise - +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.59 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.36 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.43 +0 Windows 11 X64 AMD Ryzen 9 7950X
Slower 0.67 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.65 +0 debian 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.50 +0 ubuntu 18.04 X64 AMD Ryzen 9 5900X
Slower 0.21 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.53 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.77 +0 ubuntu 20.04 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Noise - +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.64 +0 macOS Big Sur 11.7 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 0.66 +0 macOS Monterey 12.6 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Numerics.Tests.Perf_Matrix4x4.SubtractOperatorBenchmark

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Same 1.00 +0 ubuntu 18.04 Arm64 Unknown processor
Same 0.96 +0 Windows 11 Arm64 Unknown processor
Same 0.99 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.06 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.00 +0 macOS Monterey 12.6 Arm64 Apple M1
Same 1.00 +0 macOS Monterey 12.6 Arm64 Apple M1 Max
Slower 0.46 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.61 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.68 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.65 +0 Windows 11 X64 AMD Ryzen 9 7950X
Same 0.91 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise - +0 debian 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.73 +0 ubuntu 18.04 X64 AMD Ryzen 9 5900X
Slower 0.25 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.70 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Same 0.96 +0 ubuntu 20.04 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Same 0.97 +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.83 +0 macOS Big Sur 11.7 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 0.78 +0 macOS Monterey 12.6 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Numerics.Tests.Perf_Matrix4x4.Transpose

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Same 0.97 +0 ubuntu 18.04 Arm64 Unknown processor
Same 0.92 +0 Windows 11 Arm64 Unknown processor
Same 0.98 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.03 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 0.92 +0 macOS Monterey 12.6 Arm64 Apple M1
Same 0.93 +0 macOS Monterey 12.6 Arm64 Apple M1 Max
Slower 0.76 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.64 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.45 +0 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.49 +0 Windows 11 X64 AMD Ryzen 9 7950X
Slower 0.66 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.71 +0 debian 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.19 +0 ubuntu 18.04 X64 AMD Ryzen 9 5900X
Slower 0.24 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.56 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.68 +0 ubuntu 20.04 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Same 0.89 +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.73 +0 macOS Big Sur 11.7 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell) bimodal
Slower 0.66 +0 macOS Monterey 12.6 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

This was referenced Oct 14, 2022
Closed
Open
@dakersnar dakersnar mentioned this issue Nov 29, 2022
Closed
9 tasks
@dakersnar
Copy link
Contributor

The .NET 7 GA report is also showing potential regression in the following. I ran into the same problem where the historical data does not go back far enough to help triage this. @DrewScoggins is this something we can change?

System.Numerics.Tests.Perf_Matrix4x4.AddOperatorBenchmark

Result Ratio Alloc Delta Operating System Bit Processor Name Modality
Slower 0.76 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.01 +0 macOS 13.0 Arm64 Apple M1
Slower 0.69 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Noise - +0 Windows 10 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same 1.08 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.63 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 0.93 +0 Windows 11 X64 11th Gen Intel Core i9-11900 2.50GHz
Slower 0.16 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.88 +0 raspbian 11 Arm ARMv7 Processor rev 3 (v7l)
Slower 0.69 +0 macOS Monterey 12.6 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)

@DrewScoggins
Copy link
Member

DrewScoggins commented Dec 5, 2022 •
edited
Loading

We should figure out something that we can do. Naively, we could just extend the window of that history page from 12 to 18 months. However, that would not be feasible with the way those pages are currently generated, they would take too long to retrieve all of the data.

For these individual tests that we need more data for we can go ahead and generate longer trends, and just do them on a case by case basis, just get me a list of the tests that we need this for. However, I think the right way to fix this going forward is to update the generation method for the data on these pages so that we are not redoing work over and over (Currently, we regenerate the data every day, even though all we need to do is drop the earliest and grab the latest)

@dakersnar dakersnar mentioned this issue Dec 5, 2022
Closed
9 tasks
@tannergooding
Copy link
Member

This was resolved with my rewrite of Matrix4x4 so be able to utilize promotion, field enregistration, and general SIMD/vectorization

@ghost ghost locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
area-System.Numerics tenet-performance Performance related issue
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
5 participants
@adamsitnik @DrewScoggins @tannergooding @jozkee @dakersnar