Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.NET 7.0 Preview 5 Microbenchmarks Performance Study Report #70590

Closed
14 tasks done
dakersnar opened this issue Jun 10, 2022 · 6 comments
Closed
14 tasks done

.NET 7.0 Preview 5 Microbenchmarks Performance Study Report #70590

dakersnar opened this issue Jun 10, 2022 · 6 comments
Labels
area-Meta tenet-performance Performance related issue tracking This issue is tracking the completion of other related issues.
Milestone

Comments

@dakersnar
Copy link
Contributor

dakersnar commented Jun 10, 2022

Data

This time we have covered following configs:

  • architectures: arm64, x64, x86
  • Unix: ubuntu 18.04, ubuntu 20.04, debian 11, macOS Monterey 12.3, macOS Big Sur 11.6.6
  • Windows: 10 & 11
Operating System Bit Processor Name
debian 11 Arm64 Unknown processor
ubuntu 18.04 Arm64 Unknown processor
ubuntu 20.04 Arm64 Unknown processor
Windows 11 Arm64 Microsoft SQ1 3.0 GHz
macOS Monterey 12.3 Arm64 Apple M1 Max
Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Windows 11 X64 AMD Ryzen 9 5900X
Windows 11 X64 AMD Ryzen 9 5950X
Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report!

The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here.

Notable Improvements

  • System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE, System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE
  • System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
  • PerfLabTests.CastingPerf.IFooFooIsIFoo, PerfLabTests.CastingPerf.ObjFooIsObj2, PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo
  • System.Memory.Span<Byte>.Reverse(Size: 512), System.Memory.Span<Char>.Reverse(Size: 512), System.Memory.Span<Int32>.Reverse(Size: 512), System.Tests.Perf_Array.Reverse
  • System.Memory.Span<Byte>.Fill(Size: 512), System.Memory.Span<Byte>.Clear(Size: 512)

Regressions

By design

  • System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark, System.Numerics.Tests.Perf_Vector3.EqualsBenchmark, System.Numerics.Tests.Perf_Vector2.EqualsBenchmark, System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark, System.Numerics.Tests.Perf_Vector4.EqualsBenchmark

Investigation in progress

Already solved

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

  • System.Text.RegularExpressions.Tests.*.Count(*, Options: None)
    • This was an interesting case. There was a pattern of regression for a lot of these Options: None regex tests, but historical data indicates it is probably noise. These are worth keeping an eye on.
  • BenchmarksGame.RegexRedux_5.RunBench(options: None)
  • System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong
  • System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!"), System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")
    • Trimodal
  • System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
  • System.Net.Security.Tests.SslStreamTests.ConcurrentReadWrite
  • System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
  • PerfLabTests.CastingPerf.CheckListIsVariantGenericInterface
  • System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)
  • System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmark
  • System.Numerics.Tests.Perf_Vector2.ReflectBenchmark
  • System.Numerics.Tests.Perf_VectorOf<Double>.NegateBenchmark
  • System.Numerics.Tests.Perf_VectorOf<Int32>.XorBenchmark
  • PerfLabTests.LowLevelPerf.GenericGenericMethod
  • PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod
  • System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512)
  • System.Collections.IterateForEach<Int32>.LinkedList(Size: 512)
  • System.Memory.Span<Byte>.SequenceCompareTo(Size: 512)

Statistics

Total: 81617
Same: 79.67 %
Slower: 7.73 %
Faster: 4.27 %
Noise: 8.27 %
Unknown: 0.05 %

Statistics per Architecture

Architecture Same Slower Faster Noise Unknown
Arm64 80.38 % 6.75 % 4.06 % 8.72 % 0.09 %
X64 78.62 % 8.60 % 4.33 % 8.41 % 0.04 %
X86 88.77 % 2.20 % 4.64 % 4.39 % 0.00 %

Statistics per Operating System

Operating System Same Slower Faster Noise Unknown
debian 11 73.40 % 8.72 % 5.04 % 12.80 % 0.05 %
macOS Monterey 12.3 76.73 % 14.64 % 2.86 % 5.63 % 0.13 %
ubuntu 18.04 68.59 % 20.18 % 2.39 % 8.74 % 0.10 %
ubuntu 20.04 81.91 % 5.18 % 4.71 % 8.17 % 0.03 %
Windows 11 83.42 % 3.64 % 4.29 % 8.60 % 0.05 %
macOS Big Sur 11.6.6 89.05 % 3.05 % 3.62 % 4.21 % 0.07 %
Windows 10 81.18 % 5.00 % 5.77 % 8.05 % 0.00 %

Statistics per Namespace

Namespace Same Slower Faster Noise Unknown
BenchmarksGame 78.31 % 12.17 % 9.52 % 0.00 % 0.00 %
Benchstone.BenchF 88.89 % 3.89 % 7.22 % 0.00 % 0.00 %
Benchstone.BenchI 84.72 % 6.71 % 8.56 % 0.00 % 0.00 %
Benchstone.MDBenchF 93.33 % 0.00 % 6.67 % 0.00 % 0.00 %
Benchstone.MDBenchI 90.56 % 3.89 % 5.56 % 0.00 % 0.00 %
Devirtualization 83.33 % 1.11 % 1.11 % 14.44 % 0.00 %
Exceptions 89.35 % 4.63 % 6.02 % 0.00 % 0.00 %
FractalPerf 83.33 % 11.11 % 5.56 % 0.00 % 0.00 %
GuardedDevirtualization 85.19 % 0.13 % 0.26 % 14.42 % 0.00 %
Inlining 51.85 % 1.85 % 0.00 % 46.30 % 0.00 %
Interop 83.33 % 3.70 % 12.96 % 0.00 % 0.00 %
JetStream 94.44 % 2.78 % 2.78 % 0.00 % 0.00 %
Layout 97.22 % 2.78 % 0.00 % 0.00 % 0.00 %
Lowering 88.89 % 0.00 % 0.00 % 11.11 % 0.00 %
MicroBenchmarks.Serializers 88.52 % 9.87 % 1.62 % 0.00 % 0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance 84.72 % 13.19 % 2.08 % 0.00 % 0.00 %
Microsoft.Extensions.Caching.Memory.Tests 80.30 % 11.11 % 8.59 % 0.00 % 0.00 %
Microsoft.Extensions.Configuration.Xml 94.44 % 5.56 % 0.00 % 0.00 % 0.00 %
Microsoft.Extensions.DependencyInjection 90.50 % 3.23 % 5.56 % 0.72 % 0.00 %
Microsoft.Extensions.Logging 89.37 % 6.03 % 3.17 % 1.43 % 0.00 %
Microsoft.Extensions.Primitives.Performance 62.50 % 0.00 % 0.69 % 36.81 % 0.00 %
Microsoft.Extensions.Primitives 73.77 % 3.09 % 1.23 % 21.91 % 0.00 %
PerfLabTests 64.51 % 11.11 % 19.44 % 4.93 % 0.00 %
PerfLabTests.CastingPerf2 50.37 % 4.81 % 44.81 % 0.00 % 0.00 %
SciMark2 87.78 % 5.56 % 6.67 % 0.00 % 0.00 %
SIMD 97.53 % 2.47 % 0.00 % 0.00 % 0.00 %
Span 90.44 % 7.78 % 1.78 % 0.00 % 0.00 %
StoreBlock 83.52 % 0.56 % 0.19 % 15.74 % 0.00 %
System.Buffers.Binary.Tests 59.03 % 5.56 % 26.39 % 9.03 % 0.00 %
System.Buffers.Tests 84.63 % 8.00 % 4.26 % 3.11 % 0.00 %
System.Buffers.Text.Tests 75.49 % 4.74 % 6.21 % 13.56 % 0.00 %
System.Collections 83.32 % 10.72 % 5.35 % 0.60 % 0.00 %
System.Collections.Concurrent 84.72 % 5.34 % 2.52 % 7.42 % 0.00 %
System.Collections.Tests 78.71 % 10.96 % 4.72 % 5.61 % 0.00 %
System.ComponentModel.Tests 92.93 % 5.05 % 2.02 % 0.00 % 0.00 %
System.ConsoleTests 92.59 % 1.85 % 5.56 % 0.00 % 0.00 %
System.Diagnostics 78.61 % 8.33 % 13.06 % 0.00 % 0.00 %
System.Diagnostics.Tracing 84.72 % 4.17 % 11.11 % 0.00 % 0.00 %
System.Formats.Cbor.Tests 92.31 % 7.69 % 0.00 % 0.00 % 0.00 %
System.Globalization.Tests 85.67 % 10.90 % 1.91 % 1.53 % 0.00 %
System 87.65 % 4.63 % 4.01 % 3.70 % 0.00 %
System.IO.Compression 87.19 % 10.80 % 2.01 % 0.00 % 0.00 %
System.IO.MemoryMappedFiles.Tests 88.21 % 8.96 % 2.83 % 0.00 % 0.00 %
System.IO.Pipelines.Tests 86.11 % 8.33 % 5.56 % 0.00 % 0.00 %
System.IO.Pipes.Tests 78.70 % 19.44 % 1.85 % 0.00 % 0.00 %
System.IO.Tests 84.91 % 8.61 % 4.96 % 1.53 % 0.00 %
System.Linq.Tests 81.46 % 13.59 % 4.79 % 0.17 % 0.00 %
System.MathBenchmarks 86.29 % 10.75 % 1.70 % 1.25 % 0.00 %
System.Memory 74.01 % 6.89 % 3.89 % 15.21 % 0.00 %
System.Net.Http.Tests 76.62 % 21.99 % 1.39 % 0.00 % 0.00 %
System.Net.NetworkInformation.Tests 86.11 % 8.33 % 5.56 % 0.00 % 0.00 %
System.Net.Primitives.Tests 82.32 % 6.06 % 4.04 % 7.58 % 0.00 %
System.Net.Security.Tests 74.47 % 8.87 % 16.67 % 0.00 % 0.00 %
System.Net.Sockets.Tests 77.27 % 14.77 % 7.95 % 0.00 % 0.00 %
System.Net.Tests 81.31 % 15.89 % 2.80 % 0.00 % 0.00 %
System.Net.WebSockets.Tests 72.22 % 19.44 % 8.33 % 0.00 % 0.00 %
System.Numerics.Tests 66.28 % 2.52 % 0.86 % 30.34 % 0.00 %
System.Reflection 75.93 % 2.96 % 19.81 % 1.30 % 0.00 %
System.Runtime.InteropServices.Tests 88.89 % 5.56 % 5.56 % 0.00 % 0.00 %
System.Runtime.Serialization.Formatters.Tests 100.00 % 0.00 % 0.00 % 0.00 % 0.00 %
System.Security.Cryptography.Primitives.Tests.Performance 86.11 % 9.72 % 4.17 % 0.00 % 0.00 %
System.Security.Cryptography.Tests 93.83 % 4.94 % 1.23 % 0.00 % 0.00 %
System.Security.Cryptography.X509Certificates.Tests 90.74 % 7.41 % 1.85 % 0.00 % 0.00 %
System.Tests 80.30 % 5.92 % 5.95 % 7.84 % 0.00 %
System.Text.Encodings.Web.Tests 89.35 % 4.17 % 6.02 % 0.46 % 0.00 %
System.Text.Json.Document.Tests 91.67 % 5.44 % 2.89 % 0.00 % 0.00 %
System.Text.Json.Reader.Tests 69.44 % 6.94 % 23.61 % 0.00 % 0.00 %
System.Text.Json.Serialization.Tests 93.23 % 4.53 % 2.24 % 0.00 % 0.00 %
System.Text.Json.Tests 91.08 % 5.38 % 3.40 % 0.14 % 0.00 %
System.Text 67.41 % 32.59 % 0.00 % 0.00 % 0.00 %
System.Text.RegularExpressions.Tests 74.11 % 20.43 % 4.63 % 0.00 % 0.84 %
System.Text.Tests 85.43 % 7.28 % 5.60 % 1.68 % 0.00 %
System.Threading.Channels.Tests 91.18 % 7.84 % 0.98 % 0.00 % 0.00 %
System.Threading.Tasks.Tests 86.27 % 13.73 % 0.00 % 0.00 % 0.00 %
System.Threading.Tasks 84.45 % 8.82 % 6.72 % 0.00 % 0.00 %
System.Threading.Tests 83.23 % 8.76 % 1.75 % 6.26 % 0.00 %
System.Xml.Linq 82.66 % 4.64 % 4.95 % 7.74 % 0.00 %
System.Xml.Tests 95.29 % 4.71 % 0.00 % 0.00 % 0.00 %
V8.Crypto 94.12 % 5.88 % 0.00 % 0.00 % 0.00 %
V8.Richards 88.24 % 5.88 % 5.88 % 0.00 % 0.00 %
XmlDocumentTests.XmlDocumentTests 86.27 % 3.92 % 3.92 % 5.88 % 0.00 %
XmlDocumentTests.XmlNodeListTests 94.12 % 2.94 % 2.94 % 0.00 % 0.00 %
XmlDocumentTests.XmlNodeTests 53.33 % 0.00 % 0.00 % 46.67 % 0.00 %
System.Formats.Tar.Tests 96.88 % 3.12 % 0.00 % 0.00 % 0.00 %
System.Drawing.Tests 92.50 % 2.50 % 2.50 % 2.50 % 0.00 %
HardwareIntrinsics.RayTracer 76.92 % 0.00 % 15.38 % 7.69 % 0.00 %

Big thanks to everyone involved!

@dakersnar dakersnar added area-Meta tenet-performance Performance related issue tracking This issue is tracking the completion of other related issues. labels Jun 10, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 10, 2022
@ghost
Copy link

ghost commented Jun 10, 2022

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

Data

This time we have covered following configs:

  • architectures: arm64, x64, x86
  • Unix: ubuntu 18.04, ubuntu 20.04, debian 11, macOS Monterey 12.3, macOS Big Sur 11.6.6
  • Windows: 10 & 11
Operating System Bit Processor Name
debian 11 Arm64 Unknown processor
ubuntu 18.04 Arm64 Unknown processor
ubuntu 20.04 Arm64 Unknown processor
Windows 11 Arm64 Microsoft SQ1 3.0 GHz
macOS Monterey 12.3 Arm64 Apple M1 Max
Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Windows 11 X64 AMD Ryzen 9 5900X
Windows 11 X64 AMD Ryzen 9 5950X
Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report!

The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here.

Notable Improvements

  • System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE, System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE
  • System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled), System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
  • PerfLabTests.CastingPerf.IFooFooIsIFoo, PerfLabTests.CastingPerf.ObjFooIsObj2, PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo
  • System.Memory.Span<Byte>.Reverse(Size: 512), System.Memory.Span<Char>.Reverse(Size: 512), System.Memory.Span<Int32>.Reverse(Size: 512), System.Tests.Perf_Array.Reverse
  • System.Memory.Span<Byte>.Fill(Size: 512), System.Memory.Span<Byte>.Clear(Size: 512)

Regressions

By design

  • System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark, System.Numerics.Tests.Perf_Vector3.EqualsBenchmark, System.Numerics.Tests.Perf_Vector2.EqualsBenchmark, System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark, System.Numerics.Tests.Perf_Vector4.EqualsBenchmark

Investigation in progress

Already solved

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

  • System.Text.RegularExpressions.Tests.*.Count(*, Options: None)
    • This was an interesting case. There was a pattern of regression for a lot of these Options: None regex tests, but historical data indicates it is probably noise. These are worth keeping an eye on.
  • BenchmarksGame.RegexRedux_5.RunBench(options: None)
  • System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong
  • System.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!"), System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")
    • Trimodal
  • System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)
  • System.Net.Security.Tests.SslStreamTests.ConcurrentReadWrite
  • System.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmark
  • PerfLabTests.CastingPerf.CheckListIsVariantGenericInterface
  • System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)
  • System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmark
  • System.Numerics.Tests.Perf_Vector2.ReflectBenchmark
  • System.Numerics.Tests.Perf_VectorOf<Double>.NegateBenchmark
  • System.Numerics.Tests.Perf_VectorOf<Int32>.XorBenchmark
  • PerfLabTests.LowLevelPerf.GenericGenericMethod
  • PerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethod
  • System.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512)
  • System.Collections.IterateForEach<Int32>.LinkedList(Size: 512)
  • System.Memory.Span<Byte>.SequenceCompareTo(Size: 512)

Statistics

Total: 81617
Same: 79.67 %
Slower: 7.73 %
Faster: 4.27 %
Noise: 8.27 %
Unknown: 0.05 %

Statistics per Architecture

Architecture Same Slower Faster Noise Unknown
Arm64 80.38 % 6.75 % 4.06 % 8.72 % 0.09 %
X64 78.62 % 8.60 % 4.33 % 8.41 % 0.04 %
X86 88.77 % 2.20 % 4.64 % 4.39 % 0.00 %

Statistics per Operating System

Operating System Same Slower Faster Noise Unknown
debian 11 73.40 % 8.72 % 5.04 % 12.80 % 0.05 %
macOS Monterey 12.3 76.73 % 14.64 % 2.86 % 5.63 % 0.13 %
ubuntu 18.04 68.59 % 20.18 % 2.39 % 8.74 % 0.10 %
ubuntu 20.04 81.91 % 5.18 % 4.71 % 8.17 % 0.03 %
Windows 11 83.42 % 3.64 % 4.29 % 8.60 % 0.05 %
macOS Big Sur 11.6.6 89.05 % 3.05 % 3.62 % 4.21 % 0.07 %
Windows 10 81.18 % 5.00 % 5.77 % 8.05 % 0.00 %

Statistics per Namespace

Namespace Same Slower Faster Noise Unknown
BenchmarksGame 78.31 % 12.17 % 9.52 % 0.00 % 0.00 %
Benchstone.BenchF 88.89 % 3.89 % 7.22 % 0.00 % 0.00 %
Benchstone.BenchI 84.72 % 6.71 % 8.56 % 0.00 % 0.00 %
Benchstone.MDBenchF 93.33 % 0.00 % 6.67 % 0.00 % 0.00 %
Benchstone.MDBenchI 90.56 % 3.89 % 5.56 % 0.00 % 0.00 %
Devirtualization 83.33 % 1.11 % 1.11 % 14.44 % 0.00 %
Exceptions 89.35 % 4.63 % 6.02 % 0.00 % 0.00 %
FractalPerf 83.33 % 11.11 % 5.56 % 0.00 % 0.00 %
GuardedDevirtualization 85.19 % 0.13 % 0.26 % 14.42 % 0.00 %
Inlining 51.85 % 1.85 % 0.00 % 46.30 % 0.00 %
Interop 83.33 % 3.70 % 12.96 % 0.00 % 0.00 %
JetStream 94.44 % 2.78 % 2.78 % 0.00 % 0.00 %
Layout 97.22 % 2.78 % 0.00 % 0.00 % 0.00 %
Lowering 88.89 % 0.00 % 0.00 % 11.11 % 0.00 %
MicroBenchmarks.Serializers 88.52 % 9.87 % 1.62 % 0.00 % 0.00 %
Microsoft.AspNetCore.Server.Kestrel.Performance 84.72 % 13.19 % 2.08 % 0.00 % 0.00 %
Microsoft.Extensions.Caching.Memory.Tests 80.30 % 11.11 % 8.59 % 0.00 % 0.00 %
Microsoft.Extensions.Configuration.Xml 94.44 % 5.56 % 0.00 % 0.00 % 0.00 %
Microsoft.Extensions.DependencyInjection 90.50 % 3.23 % 5.56 % 0.72 % 0.00 %
Microsoft.Extensions.Logging 89.37 % 6.03 % 3.17 % 1.43 % 0.00 %
Microsoft.Extensions.Primitives.Performance 62.50 % 0.00 % 0.69 % 36.81 % 0.00 %
Microsoft.Extensions.Primitives 73.77 % 3.09 % 1.23 % 21.91 % 0.00 %
PerfLabTests 64.51 % 11.11 % 19.44 % 4.93 % 0.00 %
PerfLabTests.CastingPerf2 50.37 % 4.81 % 44.81 % 0.00 % 0.00 %
SciMark2 87.78 % 5.56 % 6.67 % 0.00 % 0.00 %
SIMD 97.53 % 2.47 % 0.00 % 0.00 % 0.00 %
Span 90.44 % 7.78 % 1.78 % 0.00 % 0.00 %
StoreBlock 83.52 % 0.56 % 0.19 % 15.74 % 0.00 %
System.Buffers.Binary.Tests 59.03 % 5.56 % 26.39 % 9.03 % 0.00 %
System.Buffers.Tests 84.63 % 8.00 % 4.26 % 3.11 % 0.00 %
System.Buffers.Text.Tests 75.49 % 4.74 % 6.21 % 13.56 % 0.00 %
System.Collections 83.32 % 10.72 % 5.35 % 0.60 % 0.00 %
System.Collections.Concurrent 84.72 % 5.34 % 2.52 % 7.42 % 0.00 %
System.Collections.Tests 78.71 % 10.96 % 4.72 % 5.61 % 0.00 %
System.ComponentModel.Tests 92.93 % 5.05 % 2.02 % 0.00 % 0.00 %
System.ConsoleTests 92.59 % 1.85 % 5.56 % 0.00 % 0.00 %
System.Diagnostics 78.61 % 8.33 % 13.06 % 0.00 % 0.00 %
System.Diagnostics.Tracing 84.72 % 4.17 % 11.11 % 0.00 % 0.00 %
System.Formats.Cbor.Tests 92.31 % 7.69 % 0.00 % 0.00 % 0.00 %
System.Globalization.Tests 85.67 % 10.90 % 1.91 % 1.53 % 0.00 %
System 87.65 % 4.63 % 4.01 % 3.70 % 0.00 %
System.IO.Compression 87.19 % 10.80 % 2.01 % 0.00 % 0.00 %
System.IO.MemoryMappedFiles.Tests 88.21 % 8.96 % 2.83 % 0.00 % 0.00 %
System.IO.Pipelines.Tests 86.11 % 8.33 % 5.56 % 0.00 % 0.00 %
System.IO.Pipes.Tests 78.70 % 19.44 % 1.85 % 0.00 % 0.00 %
System.IO.Tests 84.91 % 8.61 % 4.96 % 1.53 % 0.00 %
System.Linq.Tests 81.46 % 13.59 % 4.79 % 0.17 % 0.00 %
System.MathBenchmarks 86.29 % 10.75 % 1.70 % 1.25 % 0.00 %
System.Memory 74.01 % 6.89 % 3.89 % 15.21 % 0.00 %
System.Net.Http.Tests 76.62 % 21.99 % 1.39 % 0.00 % 0.00 %
System.Net.NetworkInformation.Tests 86.11 % 8.33 % 5.56 % 0.00 % 0.00 %
System.Net.Primitives.Tests 82.32 % 6.06 % 4.04 % 7.58 % 0.00 %
System.Net.Security.Tests 74.47 % 8.87 % 16.67 % 0.00 % 0.00 %
System.Net.Sockets.Tests 77.27 % 14.77 % 7.95 % 0.00 % 0.00 %
System.Net.Tests 81.31 % 15.89 % 2.80 % 0.00 % 0.00 %
System.Net.WebSockets.Tests 72.22 % 19.44 % 8.33 % 0.00 % 0.00 %
System.Numerics.Tests 66.28 % 2.52 % 0.86 % 30.34 % 0.00 %
System.Reflection 75.93 % 2.96 % 19.81 % 1.30 % 0.00 %
System.Runtime.InteropServices.Tests 88.89 % 5.56 % 5.56 % 0.00 % 0.00 %
System.Runtime.Serialization.Formatters.Tests 100.00 % 0.00 % 0.00 % 0.00 % 0.00 %
System.Security.Cryptography.Primitives.Tests.Performance 86.11 % 9.72 % 4.17 % 0.00 % 0.00 %
System.Security.Cryptography.Tests 93.83 % 4.94 % 1.23 % 0.00 % 0.00 %
System.Security.Cryptography.X509Certificates.Tests 90.74 % 7.41 % 1.85 % 0.00 % 0.00 %
System.Tests 80.30 % 5.92 % 5.95 % 7.84 % 0.00 %
System.Text.Encodings.Web.Tests 89.35 % 4.17 % 6.02 % 0.46 % 0.00 %
System.Text.Json.Document.Tests 91.67 % 5.44 % 2.89 % 0.00 % 0.00 %
System.Text.Json.Reader.Tests 69.44 % 6.94 % 23.61 % 0.00 % 0.00 %
System.Text.Json.Serialization.Tests 93.23 % 4.53 % 2.24 % 0.00 % 0.00 %
System.Text.Json.Tests 91.08 % 5.38 % 3.40 % 0.14 % 0.00 %
System.Text 67.41 % 32.59 % 0.00 % 0.00 % 0.00 %
System.Text.RegularExpressions.Tests 74.11 % 20.43 % 4.63 % 0.00 % 0.84 %
System.Text.Tests 85.43 % 7.28 % 5.60 % 1.68 % 0.00 %
System.Threading.Channels.Tests 91.18 % 7.84 % 0.98 % 0.00 % 0.00 %
System.Threading.Tasks.Tests 86.27 % 13.73 % 0.00 % 0.00 % 0.00 %
System.Threading.Tasks 84.45 % 8.82 % 6.72 % 0.00 % 0.00 %
System.Threading.Tests 83.23 % 8.76 % 1.75 % 6.26 % 0.00 %
System.Xml.Linq 82.66 % 4.64 % 4.95 % 7.74 % 0.00 %
System.Xml.Tests 95.29 % 4.71 % 0.00 % 0.00 % 0.00 %
V8.Crypto 94.12 % 5.88 % 0.00 % 0.00 % 0.00 %
V8.Richards 88.24 % 5.88 % 5.88 % 0.00 % 0.00 %
XmlDocumentTests.XmlDocumentTests 86.27 % 3.92 % 3.92 % 5.88 % 0.00 %
XmlDocumentTests.XmlNodeListTests 94.12 % 2.94 % 2.94 % 0.00 % 0.00 %
XmlDocumentTests.XmlNodeTests 53.33 % 0.00 % 0.00 % 46.67 % 0.00 %
System.Formats.Tar.Tests 96.88 % 3.12 % 0.00 % 0.00 % 0.00 %
System.Drawing.Tests 92.50 % 2.50 % 2.50 % 2.50 % 0.00 %
HardwareIntrinsics.RayTracer 76.92 % 0.00 % 15.38 % 7.69 % 0.00 %

Big thanks to everyone involved!

Author: dakersnar
Assignees: -
Labels:

area-Meta, tenet-performance, tracking

Milestone: -

@jeffhandley jeffhandley added this to the 7.0.0 milestone Jun 10, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jun 10, 2022
@TonyValenti
Copy link

Am I reading this correctly?
It seems like a lot of the tests run slower now.

@dakersnar
Copy link
Contributor Author

@TonyValenti That's correct. One of the purposes of this report is to provide an additional line of defense against unintended regressions that can slip past us. We analyze the data and try to track down what might be causing specific regressions. Sometimes, the reported regression ends up just being a noisy benchmark. Sometimes, there is an actual regression, and the report helps notify the right people about the problem, while providing data to help them solve it.

It seems like a lot of the tests run slower now.

If you happen to be looking at the full report generated by the tool when you come to this conclusion, I want to clarify that the data is sorted from most regressed to most improved. Only looking at the top of the file will lead to that conclusion, but the whole report is 100k lines long, and most benchmarks in the middle end up not changing on a month-to-month basis.

Let me know if that cleared things up!

@AndyAyersMS
Copy link
Member

Also worth noting that some regressions are from a bug fix (#68691). If you factor those out it won't look quite as bad.

@dakersnar
Copy link
Contributor Author

There was also a particularly high number of noisy benchmarks this month, a lot of which were incorrectly flagged as regressions by the tool. This unfortunately skews the overall statistics. You can see specifics in my Noise, flaky or multimodal section, where I manually investigated and triaged benchmarks that were flagged as regressions but ended up being noise.

@danmoseley
Copy link
Member

Am I reading this correctly? It seems like a lot of the tests run slower now.

On top of what others said, # tests does not necessarily correlate well with # distinct regressions, # apps affected, etc. often a regression in many tests is caused by one or two regressions, which can then get fixed.

Also, this is a point in time during development -- the key thing is the system spots regressions in time to fix them.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Meta tenet-performance Performance related issue tracking This issue is tracking the completion of other related issues.
Projects
None yet
Development

No branches or pull requests

5 participants