System.Text.Json source generators are not improving deserialize performance

[cirrusone]

I've been attempting to apply some of the examples in the blog post https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/

However, I'm not seeing any performance improvements. In fact the source generator tests are always marginally slower than using SystemTextJson without source generators.

Have a made a code error or have I mistaken expected performance improvement of using source generators for deserializing json string data?

Below is sample benchmarks and code to replicate:

BenchmarkDotNet=v0.13.5, OS=Windows 11 (10.0.22621.1265/22H2/2022Update/SunValley2)
12th Gen Intel Core i9-12900H, 1 CPU, 20 logical and 14 physical cores
.NET SDK=7.0.103
[Host] : .NET 7.0.3 (7.0.323.6910), X64 RyuJIT AVX2
DefaultJob : .NET 7.0.3 (7.0.323.6910), X64 RyuJIT AVX2

Method	Mean	Error	StdDev
SystemTextJson	258.2 ns	1.00 ns	0.89 ns
SystemTextJsonMetadataSrcGen	262.8 ns	1.21 ns	1.08 ns
SystemTextJsonDefaultSrcGen	262.0 ns	1.80 ns	1.68 ns

SourceGenTest.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>disable</ImplicitUsings>
    <Nullable>disable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.5" />
  </ItemGroup>

</Project>

Program.cs:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace SourceGenTest;

internal static class Program
{
    static void Main(string[] args)
    {

#if DEBUG
        new JsonDeserializeCompare().DebugTest();
#else
        var summary = BenchmarkRunner.Run<JsonDeserializeCompare>();
#endif

    }
}

public class JsonDeserializeCompare
{
    private string _jsonToDeserialize;
    private MetadataJsonClassContext _metadataJsonClassContext;
    private DefaultJsonClassContext _defaultJsonClassContext;
    private SerializationJsonClassContext _serializationJsonClassContext;
    public JsonDeserializeCompare()
    {
        _jsonToDeserialize = "{\"Forename\":\"John\",\"Surname\":\"Smith\",\"Age\":42,\"Active\":true}";
        _metadataJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = MetadataJsonClassContext.Default });
        _defaultJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = DefaultJsonClassContext.Default });
        _serializationJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = SerializationJsonClassContext.Default });
    }

    public void DebugTest()
    {
        JsonClass jsonClass1 = SystemTextJson();
        JsonClass jsonClass2 = SystemTextJsonMetadataSrcGen();
        JsonClass jsonClass3 = SystemTextJsonDefaultSrcGen();
        //JsonClass jsonClass4 = SystemTextJsonSerializationSrcGen();
    }

    [Benchmark]
    public JsonClass SystemTextJson()
    {
        return System.Text.Json.JsonSerializer.Deserialize<JsonClass>(_jsonToDeserialize);
    }

    [Benchmark]
    public JsonClass SystemTextJsonMetadataSrcGen()
    {
        return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _metadataJsonClassContext.JsonClass);
    }

    [Benchmark]
    public JsonClass SystemTextJsonDefaultSrcGen()
    {
        return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _defaultJsonClassContext.JsonClass);
    }

    //[Benchmark]
    //public JsonClass SystemTextJsonSerializationSrcGen()
    //{
    //    return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _serializationJsonClassContext.JsonClass);
    //}
}

public class JsonClass
{
    public string Forename { get; set; }

    public string Surname { get; set; }

    public int Age { get; set; }

    public bool Active { get; set; }
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Metadata)]
[JsonSerializable(typeof(JsonClass))]
internal partial class MetadataJsonClassContext : JsonSerializerContext
{
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(JsonClass))]
internal partial class DefaultJsonClassContext : JsonSerializerContext
{
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Serialization)]
[JsonSerializable(typeof(JsonClass))]
internal partial class SerializationJsonClassContext : JsonSerializerContext
{
}

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I've been attempting to apply some of the examples in the blog post https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/

However, I'm not seeing any performance improvements. In fact the source generator tests are always marginally slower than using SystemTextJson without source generators.

Have a made a code error or have I mistaken expected performance improvement of using source generators for deserializing json string data?

Below is sample benchmarks and code to replicate:

BenchmarkDotNet=v0.13.5, OS=Windows 11 (10.0.22621.1265/22H2/2022Update/SunValley2)
12th Gen Intel Core i9-12900H, 1 CPU, 20 logical and 14 physical cores
.NET SDK=7.0.103
[Host] : .NET 7.0.3 (7.0.323.6910), X64 RyuJIT AVX2
DefaultJob : .NET 7.0.3 (7.0.323.6910), X64 RyuJIT AVX2

Method	Mean	Error	StdDev
SystemTextJson	258.2 ns	1.00 ns	0.89 ns
SystemTextJsonMetadataSrcGen	262.8 ns	1.21 ns	1.08 ns
SystemTextJsonDefaultSrcGen	262.0 ns	1.80 ns	1.68 ns

SourceGenTest.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>disable</ImplicitUsings>
    <Nullable>disable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.5" />
  </ItemGroup>

</Project>

Program.cs:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace SourceGenTest;

internal static class Program
{
    static void Main(string[] args)
    {

#if DEBUG
        new JsonDeserializeCompare().DebugTest();
#else
        var summary = BenchmarkRunner.Run<JsonDeserializeCompare>();
#endif

    }
}

public class JsonDeserializeCompare
{
    private string _jsonToDeserialize;
    private MetadataJsonClassContext _metadataJsonClassContext;
    private DefaultJsonClassContext _defaultJsonClassContext;
    private SerializationJsonClassContext _serializationJsonClassContext;
    public JsonDeserializeCompare()
    {
        _jsonToDeserialize = "{\"Forename\":\"John\",\"Surname\":\"Smith\",\"Age\":42,\"Active\":true}";
        _metadataJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = MetadataJsonClassContext.Default });
        _defaultJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = DefaultJsonClassContext.Default });
        _serializationJsonClassContext = new (new JsonSerializerOptions() { TypeInfoResolver = SerializationJsonClassContext.Default });
    }

    public void DebugTest()
    {
        JsonClass jsonClass1 = SystemTextJson();
        JsonClass jsonClass2 = SystemTextJsonMetadataSrcGen();
        JsonClass jsonClass3 = SystemTextJsonDefaultSrcGen();
        //JsonClass jsonClass4 = SystemTextJsonSerializationSrcGen();
    }

    [Benchmark]
    public JsonClass SystemTextJson()
    {
        return System.Text.Json.JsonSerializer.Deserialize<JsonClass>(_jsonToDeserialize);
    }

    [Benchmark]
    public JsonClass SystemTextJsonMetadataSrcGen()
    {
        return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _metadataJsonClassContext.JsonClass);
    }

    [Benchmark]
    public JsonClass SystemTextJsonDefaultSrcGen()
    {
        return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _defaultJsonClassContext.JsonClass);
    }

    //[Benchmark]
    //public JsonClass SystemTextJsonSerializationSrcGen()
    //{
    //    return System.Text.Json.JsonSerializer.Deserialize(_jsonToDeserialize, _serializationJsonClassContext.JsonClass);
    //}
}

public class JsonClass
{
    public string Forename { get; set; }

    public string Surname { get; set; }

    public int Age { get; set; }

    public bool Active { get; set; }
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Metadata)]
[JsonSerializable(typeof(JsonClass))]
internal partial class MetadataJsonClassContext : JsonSerializerContext
{
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Default)]
[JsonSerializable(typeof(JsonClass))]
internal partial class DefaultJsonClassContext : JsonSerializerContext
{
}

[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.Unspecified, GenerationMode = JsonSourceGenerationMode.Serialization)]
[JsonSerializable(typeof(JsonClass))]
internal partial class SerializationJsonClassContext : JsonSerializerContext
{
}

Author:	cirrusone
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[krwq]

source gen only improves serialization and trimmability of the app (since it doesn't use reflection). What article is referring to is likely fast-path and that is serialization only feature for now.

[krwq]

Duplicate of #55043

[krwq]

I'm closing this as duplicate. Please respond in the other issue.