Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x86: 19 Regressions on 6/29/2023 8:27:03 AM #19464

Open
performanceautofiler bot opened this issue Jul 4, 2023 · 1 comment
Open

[Perf] Windows/x86: 19 Regressions on 6/29/2023 8:27:03 AM #19464

performanceautofiler bot opened this issue Jul 4, 2023 · 1 comment
Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jul 4, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Remainder - Duration of single invocation 1.46 μs 1.69 μs 1.15 0.01 True
Parse - Duration of single invocation 154.05 ns 164.90 ns 1.07 0.01 False
ToStringD - Duration of single invocation 69.66 ns 73.97 ns 1.06 0.04 False
Remainder - Duration of single invocation 8.61 ms 9.46 ms 1.10 0.00 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

System.Numerics.Tests.Perf_BigInteger.Remainder(arguments: 1024,512 bits)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6852070066558733 > 1.522117938616792.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -72.09046653300956 (T) = (0 -1695.468176333426) / Math.Sqrt((67.60496046038823 / (29)) + (129.90057597681792 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.16942307521272956 = (1449.833009345231 - 1695.468176333426) / 1449.833009345231 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

System.Numerics.Tests.Perf_BigInteger.Parse(numberString: -2147483648)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 164.90068700133395 > 161.94406072516642.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -19.12842891482624 (T) = (0 -163.7634337425552) / Math.Sqrt((3.091219105517165 / (29)) + (1.499114706069037 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.057074647319094486 = (154.9213522033517 - 163.7634337425552) / 154.9213522033517 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

System.Numerics.Tests.Perf_BigInteger.ToStringD(numberString: -2147483648)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.96757204871928 > 73.23569700397475.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -29.90498683110217 (T) = (0 -73.89573484344905) / Math.Sqrt((0.13759801062762025 / (29)) + (0.18869051193942268 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.057786735233795364 = (69.85882161503602 - 73.89573484344905) / 69.85882161503602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

System.Numerics.Tests.Perf_BigInteger.Remainder(arguments: 65536,32768 bits)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.4611775 > 8.9995014375.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -127.48901023235541 (T) = (0 -9466840.734890109) / Math.Sqrt((1167329264.7722824 / (29)) + (161816497.605084 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.10733953242200389 = (8549176.162964191 - 9466840.734890109) / 8549176.162964191 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IterateTryGetTenSegments - Duration of single invocation 46.67 ns 55.95 ns 1.20 0.00 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Buffers.Tests.ReadOnlySequenceTests<Byte>.IterateTryGetTenSegments

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.95043633258059 > 48.762923476881326.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -56.77559976544997 (T) = (0 -56.27510304649753) / Math.Sqrt((0.0562316394686468 / (29)) + (0.40186047562363936 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.21449023275438703 = (46.33639820953451 - 56.27510304649753) / 46.33639820953451 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EncodeUtf16 - Duration of single invocation 95.37 ns 112.73 ns 1.18 0.10 False
EncodeUtf16 - Duration of single invocation 82.02 ns 88.35 ns 1.08 0.06 False
EncodeUtf16 - Duration of single invocation 151.93 ns 167.56 ns 1.10 0.09 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,&lorem ipsum=dolor sit amet,16)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 112.7297124501802 > 97.50890099794886.
IsChangePoint: Marked as a change because one of 4/25/2023 1:57:48 PM, 5/19/2023 10:30:38 AM, 6/22/2023 7:55:14 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -26.182278719093848 (T) = (0 -111.4896132802586) / Math.Sqrt((7.076064218057058 / (29)) + (2.8009998974881563 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.18552945888634945 = (94.04204378437679 - 111.4896132802586) / 94.04204378437679 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: JavaScript,&Hello+<World>!,16)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 88.35012223327267 > 86.1827469151956.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -11.755485437701857 (T) = (0 -89.89698672818167) / Math.Sqrt((2.5756522566659554 / (29)) + (4.0850048596662045 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.08775279939312303 = (82.64468432380669 - 89.89698672818167) / 82.64468432380669 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,&lorem ipsum=dolor sit amet,512)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 167.56340096828592 > 156.6996247860263.
IsChangePoint: Marked as a change because one of 5/24/2023 6:07:01 PM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -19.30317921191681 (T) = (0 -170.06867642033308) / Math.Sqrt((10.01908416485317 / (29)) + (8.517531650997908 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.12467474955976728 = (151.21587506690557 - 170.06867642033308) / 151.21587506690557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
StaticMethod4_int_string_struct_class - Duration of single invocation 58.36 ns 63.90 ns 1.09 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'

Payloads

Baseline
Compare

System.Reflection.Invoke.StaticMethod4_int_string_struct_class

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.89558510638298 > 61.27549329661911.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 6/8/2023 9:27:53 AM, 6/22/2023 7:55:14 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -6.722963109869207 (T) = (0 -63.90837385891732) / Math.Sqrt((3.2810652615167926 / (29)) + (4.67258964536705 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.07564410341961655 = (59.41405122358227 - 63.90837385891732) / 59.41405122358227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Document.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseThenWrite - Duration of single invocation 14.58 μs 15.71 μs 1.08 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_ParseThenWrite*'

Payloads

Baseline
Compare

System.Text.Json.Document.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.70833270511371 > 15.393621096040354.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/27/2023 7:38:18 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -34.90831861211126 (T) = (0 -15634.011789574748) / Math.Sqrt((8146.3888356107345 / (29)) + (7508.088038989228 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.06818178460081896 = (14636.096603554422 - 15634.011789574748) / 14636.096603554422 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.ReadOnlySequenceTests<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IterateTryGetArray - Duration of single invocation 6.52 ns 7.67 ns 1.18 0.05 False
IterateTryGetSingleSegment - Duration of single invocation 14.93 ns 16.33 ns 1.09 0.03 False
IterateTryGetTenSegments - Duration of single invocation 55.44 ns 63.84 ns 1.15 0.02 True

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.ReadOnlySequenceTests&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateTryGetArray

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.671664448779576 > 6.846450693199556.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -10.91340544170543 (T) = (0 -7.873183627378845) / Math.Sqrt((0.009425120539659873 / (29)) + (0.18792142023752761 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.19407024750309854 = (6.593568212458467 - 7.873183627378845) / 6.593568212458467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateTryGetSingleSegment

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.32589252886242 > 15.645150877852451.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -41.786283316489516 (T) = (0 -16.173450057521492) / Math.Sqrt((0.01837072785424104 / (28)) + (0.0032043793381716793 / (14))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (28) + (14) - 2, .025) and -0.08325885962141019 = (14.930364902045644 - 16.173450057521492) / 14.930364902045644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

System.Buffers.Tests.ReadOnlySequenceTests<Char>.IterateTryGetTenSegments

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.84229526244762 > 58.235432762910385.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -103.28451037901054 (T) = (0 -63.92307894298511) / Math.Sqrt((0.12972288563219153 / (29)) + (0.034130033449209396 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.15516507254483236 = (55.33674836805995 - 63.92307894298511) / 55.33674836805995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Span.IndexerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteViaIndexer2 - Duration of single invocation 653.98 ns 1.39 μs 2.12 0.04 True
CoveredIndex3 - Duration of single invocation 1.70 μs 1.94 μs 1.14 0.01 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Span.IndexerBench*'

Payloads

Baseline
Compare

Span.IndexerBench.WriteViaIndexer2(length: 1024)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3876723057644107 > 683.7474152168412.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -162.8549496254421 (T) = (0 -1394.6337874115632) / Math.Sqrt((261.86405751454464 / (29)) + (162.45568846345466 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -1.1296191482737732 = (654.8747406511749 - 1394.6337874115632) / 654.8747406511749 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

Span.IndexerBench.CoveredIndex3(length: 1024)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9416413214744785 > 1.7770125214654573.
IsChangePoint: Marked as a change because one of 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -226.59226876124504 (T) = (0 -1940.649898665871) / Math.Sqrt((22.189171464223303 / (29)) + (6.449145629562172 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.14846534195195768 = (1689.7766330218537 - 1940.649898665871) / 1689.7766330218537 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsMatch - Duration of single invocation 402.43 ns 430.06 ns 1.07 0.10 False
IsMatch - Duration of single invocation 168.19 ns 180.24 ns 1.07 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 3, Options: None)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 430.05760623996025 > 421.8595674792985.
IsChangePoint: Marked as a change because one of 4/25/2023 8:51:41 AM, 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -7.335359666428924 (T) = (0 -437.2871662690936) / Math.Sqrt((11.39913124327737 / (29)) + (311.59436333510655 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.08676023892154659 = (402.3768542572355 - 437.2871662690936) / 402.3768542572355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_BoostDocs_Simple.IsMatch(Id: 1, Options: None)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.23557149703564 > 176.91673127224803.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -14.966829104163027 (T) = (0 -180.54645618772415) / Math.Sqrt((2.942991333466932 / (29)) + (5.9582393538603196 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.06403717855928227 = (169.68058995098835 - 180.54645618772415) / 169.68058995098835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 1.50 ms 1.60 ms 1.07 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: None)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6045796989247312 > 1.5730864231296442.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -178.31918528939886 (T) = (0 -1603167.7546343186) / Math.Sqrt((3251407.7303943946 / (29)) + (3302599.89375622 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.07022551703256852 = (1497971.9032297488 - 1603167.7546343186) / 1497971.9032297488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jul 4, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline cfae69f62414b68e4c98dd3b294154baabf58a5a
Compare d4fcdc1240ec278980b4e63dda9769b76c1c802a
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Email_IsNotMatch - Duration of single invocation 503.08 ns 592.84 ns 1.18 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: None)

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 592.8439357078598 > 531.0783041356318.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/29/2023 3:17:38 AM, 7/3/2023 9:53:34 PM falls between 6/24/2023 6:56:43 PM and 7/3/2023 9:53:34 PM.
IsRegressionStdDev: Marked as regression because -32.03578607497296 (T) = (0 -595.1170438148694) / Math.Sqrt((99.15612868618514 / (29)) + (52.885971992785294 / (14))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (14) - 2, .025) and -0.1687858511264987 = (509.17543469685563 - 595.1170438148694) / 509.17543469685563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Compare

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0 participants