Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x86: 16 Regressions in MicroBenchmarks.Serializers.Xml #74013

Closed
performanceautofiler bot opened this issue Aug 16, 2022 · 5 comments
Closed

[Perf] Windows/x86: 16 Regressions in MicroBenchmarks.Serializers.Xml #74013

performanceautofiler bot opened this issue Aug 16, 2022 · 5 comments
Assignees
@StephenMolloy
Labels
arch-x86 area-Serialization os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 1.69 μs 1.90 μs 1.13 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<SimpleStructWithProperties>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.9041324308327388 > 1.7541366392376907.
IsChangePoint: Marked as a change because one of 7/2/2022 3:52:19 AM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -15.578097441542482 (T) = (0 -1819.7866541226235) / Math.Sqrt((378.10951927206406 / (46)) + (1344.6593038595809 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.0925196706944675 = (1665.6786169953937 - 1819.7866541226235) / 1665.6786169953937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 1.86 μs 2.04 μs 1.09 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<LoginViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.038340320011675 > 1.9686621257566688.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -17.63387301879627 (T) = (0 -2074.8277861228403) / Math.Sqrt((419.9651480419419 / (46)) + (1643.2763080910083 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.1020358439795588 = (1882.7225969623955 - 2074.8277861228403) / 1882.7225969623955 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Fill - Duration of single invocation 16.45 ns 17.99 ns 1.09 0.24 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.Span&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.Fill(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.985373618745072 > 14.739788209317599.
IsChangePoint: Marked as a change because one of 6/17/2022 12:17:51 PM, 7/28/2022 3:42:13 AM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -6.960962726608799 (T) = (0 -17.211249069538134) / Math.Sqrt((0.9611150307626226 / (46)) + (2.151946244019295 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.19612202076953023 = (14.389208434157247 - 17.211249069538134) / 14.389208434157247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
PingPong - Duration of single invocation 14.46 ms 202.34 ms 13.99 0.20 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests*'

Payloads

Baseline
Compare

Histogram

System.Threading.Channels.Tests.SpscUnboundedChannelPerfTests.PingPong

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.33888684210527 > 15.199972355769233.
IsChangePoint: Marked as a change because one of 6/19/2022 1:08:59 PM, 6/20/2022 7:45:04 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -7.502974286929925 (T) = (0 -159176796.73751688) / Math.Sqrt((162483430294.9075 / (46)) + (5574842888807054 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -9.954388677251291 = (14530869.903135298 - 159176796.73751688) / 14530869.903135298 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 457.92 μs 517.87 μs 1.13 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<CollectionsOfPrimitives>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 517.868755186722 > 481.4501926605505.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -31.785404691094456 (T) = (0 -521716.46695516066) / Math.Sqrt((22110268.597736545 / (46)) + (57187403.098170385 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.1444730650637177 = (455857.3573123929 - 521716.46695516066) / 455857.3573123929 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 45.17 μs 53.72 μs 1.19 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<IndexViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.71508688297708 > 47.4845005423974.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -102.86392206532058 (T) = (0 -53927.12842656072) / Math.Sqrt((132085.95873938745 / (46)) + (71478.15011856199 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.20003222320187494 = (44938.06698179708 - 53927.12842656072) / 44938.06698179708 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 2.26 μs 2.69 μs 1.19 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.686717647985451 > 2.3651355056915104.
IsChangePoint: Marked as a change because one of 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -38.885527707505695 (T) = (0 -2697.361070222953) / Math.Sqrt((947.5388208079711 / (46)) + (1521.4540071978086 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.18941467268095924 = (2267.805444288878 - 2697.361070222953) / 2267.805444288878 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in Microsoft.Extensions.Primitives.StringSegmentBenchmark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LastIndexOf - Duration of single invocation 5.32 ns 10.22 ns 1.92 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Primitives.StringSegmentBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.StringSegmentBenchmark.LastIndexOf

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.223998107141403 > 5.5926392660145945.
IsChangePoint: Marked as a change because one of 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -10.928573812581533 (T) = (0 -10.286233134488544) / Math.Sqrt((0.012555335652936222 / (46)) + (3.0744043363811238 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.9279588916764597 = (5.335296918879911 - 10.286233134488544) / 5.335296918879911 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 833.07 μs 923.46 μs 1.11 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 923.4558425584255 > 878.3010016666667.
IsChangePoint: Marked as a change because one of 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -44.473470044218146 (T) = (0 -929248.0659105635) / Math.Sqrt((32747167.194507137 / (46)) + (54923151.128036804 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.11121943474240449 = (836241.7330524601 - 929248.0659105635) / 836241.7330524601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in System.IO.Tests.Perf_RandomAccess

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadAsync - Duration of single invocation 2.46 ms 4.27 ms 1.74 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_RandomAccess*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_RandomAccess.ReadAsync(fileSize: 1048576, bufferSize: 4096, options: Asynchronous)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.26764826388889 > 2.5560570892857144.
IsChangePoint: Marked as a change because one of 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -8.419753842172957 (T) = (0 -3925378.697961182) / Math.Sqrt((44137843417.212 / (46)) + (445929190793.15674 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.6019313304150532 = (2450403.849048969 - 3925378.697961182) / 2450403.849048969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 81.36 μs 88.74 μs 1.09 0.00 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 88.73859500235737 > 85.57505092042933.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -39.080715125483664 (T) = (0 -89290.68172956063) / Math.Sqrt((364866.53887867776 / (46)) + (464179.0516948135 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.09444941556108509 = (81585.02390335189 - 89290.68172956063) / 81585.02390335189 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 581.27 μs 641.30 μs 1.10 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 641.3007712082263 > 608.1863567246706.
IsChangePoint: Marked as a change because one of 7/6/2022 7:05:00 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -69.37476068841238 (T) = (0 -637155.1688415615) / Math.Sqrt((10904106.792671207 / (46)) + (6595039.20935352 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.09838124730026727 = (580085.6218254251 - 637155.1688415615) / 580085.6218254251 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in System.IO.Tests.Perf_FileStream

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadAsync_NoBuffering - Duration of single invocation 31.90 ms 40.81 ms 1.28 0.14 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_FileStream*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_FileStream.ReadAsync_NoBuffering(fileSize: 104857600, userBufferSize: 16384, options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.80638380952381 > 33.5646835.
IsChangePoint: Marked as a change because one of 6/17/2022 12:17:51 PM, 6/21/2022 2:34:11 AM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -9.751633879590543 (T) = (0 -40676003.641723365) / Math.Sqrt((1101355587833.9492 / (46)) + (13770201884198.176 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.30323301617832216 = (31211612.303227317 - 40676003.641723365) / 31211612.303227317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DotBenchmark - Duration of single invocation 0.30 ns 34.12 ns 113.14 0.51 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.DotBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.1193564303231 > 0.3384196981656787.
IsChangePoint: Marked as a change because one of 7/14/2022 4:09:15 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -46.23135726089836 (T) = (0 -34.104752401929055) / Math.Sqrt((8.007619116629346 / (46)) + (4.9971982332633775 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -27.929372161701004 = (1.1788970811844866 - 34.104752401929055) / 1.1788970811844866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 442.81 μs 588.18 μs 1.33 0.01 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<MyEventsListerViewModel>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 588.1822901960785 > 465.09321114092444.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -68.14421021833772 (T) = (0 -567920.4881856393) / Math.Sqrt((12184044.77248105 / (46)) + (43599957.88862694 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.27175663088924373 = (446563.81133907294 - 567920.4881856393) / 446563.81133907294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_ToStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XmlSerializer_ - Duration of single invocation 2.65 μs 2.99 μs 1.13 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_ToStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_ToStream<Location>.XmlSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.994405275836573 > 2.792300594616572.
IsChangePoint: Marked as a change because one of 7/5/2022 7:01:28 PM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -29.832186762363804 (T) = (0 -2981.386290663934) / Math.Sqrt((1015.2769407732358 / (46)) + (1206.7106319634627 / (15))) is less than -2.000995378087428 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (15) - 2, .025) and -0.112735340870809 = (2679.3309973697324 - 2981.386290663934) / 2679.3309973697324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added refs/heads/main untriaged New issue has not been triaged by the area owner labels Aug 16, 2022
@kunalspathak kunalspathak changed the title [Perf] Windows/x86: 16 Regressions on 8/13/2022 7:28:42 AM [Perf] Windows/x86: 16 Regressions in MicroBenchmarks.Serializers.Xml Aug 16, 2022
@kunalspathak kunalspathak removed refs/heads/main untriaged New issue has not been triaged by the area owner labels Aug 16, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Aug 16, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 16, 2022
@kunalspathak
Copy link
Member

bff7c3d...a2b6674

StephenMolloy

@kunalspathak kunalspathak removed their assignment Aug 16, 2022
@kunalspathak
Copy link
Member

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline e94f7ceb73244d9c4bf7a0d38fb59d01e300ef00
Compare 75228aca01a24087d21a76f32bf8d87dc9888648
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 2.23 μs 2.55 μs 1.14 0.07 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5459379423935533 > 2.3396657604634514.
IsChangePoint: Marked as a change because one of 7/2/2022 7:35:06 PM, 7/6/2022 3:55:24 AM, 8/13/2022 2:59:44 AM, 8/15/2022 12:02:22 PM falls between 8/5/2022 10:47:59 PM and 8/15/2022 12:02:22 PM.
IsRegressionStdDev: Marked as regression because -24.845524363082955 (T) = (0 -2566.4842463384825) / Math.Sqrt((1474.079066703164 / (45)) + (2635.0424949172607 / (15))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (45) + (15) - 2, .025) and -0.1624677954498125 = (2207.789545985135 - 2566.4842463384825) / 2207.789545985135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kunalspathak kunalspathak added arch-x86 os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Aug 16, 2022
@StephenMolloy StephenMolloy self-assigned this Aug 25, 2022
@StephenMolloy StephenMolloy removed the untriaged New issue has not been triaged by the area owner label Aug 25, 2022
@StephenMolloy StephenMolloy added this to the 7.0.0 milestone Aug 25, 2022
@StephenMolloy
Copy link
Member

This regression appears to be due to the addition of indent/formatting to the XmlWriter we create in 7.0. See this comment for more explanation. The regression is expected, and we intend to live with it for 7.0.

@EgorBo EgorBo mentioned this issue Sep 28, 2022
Closed
18 tasks
@ghost ghost locked as resolved and limited conversation to collaborators Oct 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@StephenMolloy StephenMolloy

Labels
arch-x86 area-Serialization os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
3 participants
@jeffschwMSFT @kunalspathak @StephenMolloy