[Perf] Linux/x64: 5 Regressions on 3/28/2023 4:36:21 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	9b38f2ab7440b551e294ff43eea20ef8866c931a
Compare	ce4857970dff1e987a3991238c8b32e8eae3cab7
Diff	Diff
Configs	AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in Burgers

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test1 - Duration of single invocation	157.97 ms	196.19 ms	1.24	0.00	True
Test2 - Duration of single invocation	157.94 ms	195.64 ms	1.24	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Burgers*'

Payloads

Baseline
Compare

Histogram

Burgers.Test1

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 196.18775016666666 > 166.1133425525.
IsChangePoint: Marked as a change because one of 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 12:39:01 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -301.35980439545324 (T) = (0 -195792688.07882783) / Math.Sqrt((535331910048.79346 / (47)) + (41317781834.43712 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.23725770716219705 = (158247297.18427253 - 195792688.07882783) / 158247297.18427253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Burgers.Test2

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.63521442857143 > 166.05170757000002.
IsChangePoint: Marked as a change because one of 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 12:39:01 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -646.3481661817668 (T) = (0 -195746645.48168498) / Math.Sqrt((20421058779.506496 / (46)) + (29423566072.951252 / (10))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (10) - 2, .025) and -0.23784952752247288 = (158134442.94272774 - 195746645.48168498) / 158134442.94272774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	9b38f2ab7440b551e294ff43eea20ef8866c931a
Compare	ce4857970dff1e987a3991238c8b32e8eae3cab7
Diff	Diff
Configs	AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in ByteMark

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
BenchAssignJagged - Duration of single invocation	1.31 secs	1.41 secs	1.08	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchAssignJagged

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4055199998666665 > 1.3712546622375001.
IsChangePoint: Marked as a change because one of 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 12:39:01 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -70.24957440675361 (T) = (0 -1403017021.5924175) / Math.Sqrt((20925520164736.586 / (47)) + (15514396538085.586 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.07613794307213557 = (1303752024.1941423 - 1403017021.5924175) / 1303752024.1941423 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	9b38f2ab7440b551e294ff43eea20ef8866c931a
Compare	ce4857970dff1e987a3991238c8b32e8eae3cab7
Diff	Diff
Configs	AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in BenchmarksGame.FannkuchRedux_5

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
RunBench - Duration of single invocation	159.42 ms	176.90 ms	1.11	0.00	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'BenchmarksGame.FannkuchRedux_5*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 176.89719993333333 > 167.74651252625.
IsChangePoint: Marked as a change because one of 3/1/2023 3:09:21 PM, 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 12:39:01 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -80.03911287670466 (T) = (0 -176965173.31190476) / Math.Sqrt((113846785421.91151 / (47)) + (427313894231.0742 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.10632724321105323 = (159957349.3266542 - 176965173.31190476) / 159957349.3266542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 18.04
Queue	TigerUbuntu
Baseline	9b38f2ab7440b551e294ff43eea20ef8866c931a
Compare	ce4857970dff1e987a3991238c8b32e8eae3cab7
Diff	Diff
Configs	AOT:true, CompilationMode:wasm, RunKind:micro

Regressions in Benchstone.MDBenchI.MDMidpoint

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Test - Duration of single invocation	446.21 ms	477.52 ms	1.07	0.00	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDMidpoint*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDMidpoint.Test

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 477.5213846923077 > 468.5312250375.
IsChangePoint: Marked as a change because one of 3/27/2023 3:21:55 PM, 3/30/2023 4:24:24 AM falls between 3/21/2023 12:39:01 PM and 3/30/2023 4:24:24 AM.
IsRegressionStdDev: Marked as regression because -198.5074117379991 (T) = (0 -477615774.69003665) / Math.Sqrt((1011148937399.0244 / (47)) + (28197840072.669678 / (10))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (47) + (10) - 2, .025) and -0.06932858643403861 = (446650151.08477914 - 477615774.69003665) / 446650151.08477914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository