Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port the incremental source generator polyfill code to the 6.0 servicing branch. #72219

Merged
merged 5 commits into from
Aug 15, 2022

Conversation

CyrusNajmabadi
Copy link
Member

@CyrusNajmabadi CyrusNajmabadi commented Jul 14, 2022

Backport of: #71653 and #71651.

Bulk of code added in #70911. However that PR also updated the RegexGenerator (which does not exist in 6.0 and thus does not need to be serviced).

Note: this is a polyfill of an API that roslyn has added. Once SDK can move to a version of roslyn containing the api, it can reference that API directly and remove the 1000 lines of added code for the polyfill.

Servicing Template:

Description:

Primary goal here is to use a more efficient incrementally built up system for determining what classes should be examined to see if we need the source generators to run. Previously, all types in a solution would be examined (costly in CPU and memory), now only types with attributes on it (that we can show could actually bind to the desired attribute) are checked. As in practice this is a tiny subset of actual types in a solution (often zero) this vastly decreases the amount of work we needed to be done.

--

Port was very clean, requiring only one deviating line between it and the code that exists in the 7.0 line.

Customer Impact:

Improved performance of 'in box' generators when in visual studio. The existing two generators cause lots of CPU and memory churn when processing. This is because each of them effectively walks all files/types looking for potential matches. This forces continual reparsing of files, and then lots of semantic binding on potential matches. For large projects (consider thousands of files being normal), this is an enormous amount of work that can use tons of CPU and memory. In Roslyn this can be particularly bad, leading to our server getting bogged down and slowing down all language services.

The new approach uses the incremental generator engine to incrementally build up a list of 'actually viable classes' to examine. The intuition generally being: if you only care about types with "JsonSerializable" on them, then only check types that have that attribute on it syntactically. This is slightly more involved than just a trivial name check because things like aliases/global-aliases mean that someone could go global using Foo = System.Text.Json.JsonSerializable; in one file and then [Foo] class MyClass in another. However, with this, we can narrow down to only the set of types that are actually relevant, not all types. Furthermore, in projects that reference the sdk but do not use the generator at all, there will be zero potential matches, and so no semantic work has to happen at all.

Regression?

Yes. This was a perf (but not functionality) regression for anyone with a large solution and who moved to .net 6. Note taht a customer doesn't need to be using the generator to have the perf hit. Just referencing the SDK is enough as it will cause Roslyn to run the generator, which does all expensive work, just to determine that is has nothing to generate.

Risk

I would consider this low. This is a non-trivial amount of code going in. However, it has been extensively tested in roslyn, with a large test suite, and has been validated for the 7.0 sdk. That said, incremental source generation is itself complex (one of the reasons that the perf issue arose in teh first place), so this is definitely more than a trivial fix.

Link the PR to the original issue and to the PR to main.

Issue PR:
#70754

Main PRs:
#71653
#71651
#70911

This was multiple PRs as i separated out the initial polyfill, and then did a single PR per generator to update.

Note any packaging impact. (If the changes touch code in a package that we ship out of band (OOB) then we will need to take care to ensure those packages ship.)

Yes. This impacts the Microsoft.Extensions.Logging.Abstractions NuGet package as well as the System.Text.Json NuGet package. Those two ship source generators that are being edited here so we will need to service those two.

Note any ref pack impact. (Touches files contained in Microsoft.NETCore.App.Ref, Microsoft.AspNetCore.App.Ref, or Microsoft.WindowsDesktop.App.Ref).

Yes. This impacts the Microsoft.NETCore.App.Ref since the Json Source generator ships in the ref pack as well, and it is being serviced by this change.

@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost assigned CyrusNajmabadi Jul 14, 2022
@@ -0,0 +1,125 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -1,7 +1,10 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,74 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,24 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,66 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,42 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,29 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,201 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -0,0 +1,400 @@
// Licensed to the .NET Foundation under one or more agreements.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

private const string JsonSourceGenerationOptionsAttributeFullName = "System.Text.Json.Serialization.JsonSourceGenerationOptionsAttribute";

internal const string JsonSerializableAttributeFullName = "System.Text.Json.Serialization.JsonSerializableAttribute";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

single deviation. In 7.0 line this was renamed (from Serializer -> Serializable). I kept the 7.0 name (more correct), requiring teh update to line 243 below.

@@ -18,7 +18,7 @@ public partial class LoggerMessageGenerator
{
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -7,6 +7,7 @@
using System.Text;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -8,6 +8,20 @@

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -11,6 +11,7 @@
using Microsoft.CodeAnalysis;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -13,4 +13,18 @@
<Compile Include="JsonSourceGenerator.Roslyn4.0.cs" />
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@@ -6,7 +6,6 @@
</PropertyGroup>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact copy of 7.0 line.

@teo-tsirpanis teo-tsirpanis added this to the 6.0.x milestone Jul 14, 2022
@joperezr
Copy link
Member

The failure in OSX seems to be related. Looks like an issue with the Logging generator:

CSC : error CS8785: Generator 'LoggerMessageGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'IndexOutOfRangeException' with message 'Index was outside the bounds of the array.' [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]CSC(0,0): error CS8785: (NETCORE_ENGINEERING_TELEMETRY=Build) Generator 'LoggerMessageGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'IndexOutOfRangeException' with message 'Index was outside the bounds of the array.'
CSC : error CS8785: Generator 'LoggerMessageGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'IndexOutOfRangeException' with message 'Index was outside the bounds of the array.' [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]CSC(0,0): error CS8785: (NETCORE_ENGINEERING_TELEMETRY=Build) Generator 'LoggerMessageGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'IndexOutOfRangeException' with message 'Index was outside the bounds of the array.'
/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(10,32): error CS8795: Partial method 'NoNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers. [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(10,32): error CS8795: (NETCORE_ENGINEERING_TELEMETRY=Build) Partial method 'NoNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers.
/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(19,36): error CS8795: Partial method 'OneLevelNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers. [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(19,36): error CS8795: (NETCORE_ENGINEERING_TELEMETRY=Build) Partial method 'OneLevelNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers.
/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(31,40): error CS8795: Partial method 'TwoLevelNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers. [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/MiscTestExtensions.cs(31,40): error CS8795: (NETCORE_ENGINEERING_TELEMETRY=Build) Partial method 'TwoLevelNamespace.CouldNotOpenSocket(ILogger, string)' must have an implementation part because it has accessibility modifiers.
/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/ArgTestExtensions.cs(11,36): error CS8795: Partial method 'ArgTestExtensions.Method1(ILogger)' must have an implementation part because it has accessibility modifiers. [/Users/runner/work/1/s/src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj]
##[error]src/libraries/Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/TestClasses/ArgTestExtensions.cs(11,36): error CS8795: (NETCORE_ENGINEERING_TELEMETRY=Build) Partial method 'ArgTestExtensions.Method1(ILogger)' must have an implementation part because it has accessibility modifiers.

@joperezr joperezr marked this pull request as ready for review July 18, 2022 19:15
@joperezr joperezr added the Servicing-consider Issue for next servicing release review label Jul 18, 2022
@joperezr
Copy link
Member

cc: @layomia @eiriktsarpalis @steveharter for the Json Source generator changes.
cc: @eerhardt @maryamariyan for the Logging generator changes.

Copy link
Member

@eiriktsarpalis eiriktsarpalis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

STJ changes LGTM.

@ghost
Copy link

ghost commented Jul 19, 2022

Tagging subscribers to this area: @dotnet/area-extensions-logging
See info in area-owners.md if you want to be subscribed.

Issue Details

Draft until i make sure i didn't break anything.

Backport of: #71653 and #71651.

Bulk of code added in #70911. However that PR also updated the RegexGenerator (which does not exist in 6.0 and thus does not need to be serviced).

Note: this is a polyfill of an API that roslyn has added. Once SDK can move to a version of roslyn containing the api, it can reference that API directly and remove the 1000 lines of added code for the polyfill.

Servicing Template:

Description:

Primary goal here is to use a more efficient incrementally built up system for determining what classes should be examined to see if we need the source generators to run. Previously, all types in a solution would be examined (costly in CPU and memory), now only types with attributes on it (that we can show could actually bind to the desired attribute) are checked. As in practice this is a tiny subset of actual types in a solution (often zero) this vastly decreases the amount of work we needed to be done.

--

Port was very clean, requiring only one deviating line between it and the code that exists in the 7.0 line.

Customer Impact:

Improved performance of 'in box' generators when in visual studio. The existing two generators cause lots of CPU and memory churn when processing. This is because each of them effectively walks all files/types looking for potential matches. This forces continual reparsing of files, and then lots of semantic binding on potential matches. For large projects (consider thousands of files being normal), this is an enormous amount of work that can use tons of CPU and memory. In Roslyn this can be particularly bad, leading to our server getting bogged down and slowing down all language services.

The new approach uses the incremental generator engine to incrementally build up a list of 'actually viable classes' to examine. The intuition generally being: if you only care about types with "JsonSerializable" on them, then only check types that have that attribute on it syntactically. This is slightly more involved than just a trivial name check because things like aliases/global-aliases mean that someone could go global using Foo = System.Text.Json.JsonSerializable; in one file and then [Foo] class MyClass in another. However, with this, we can narrow down to only the set of types that are actually relevant, not all types. Furthermore, in projects that reference the sdk but do not use the generator at all, there will be zero potential matches, and so no semantic work has to happen at all.

Regression?

Yes. This was a perf (but not functionality) regression for anyone with a large solution and who moved to .net 6. Note taht a customer doesn't need to be using the generator to have the perf hit. Just referencing the SDK is enough as it will cause Roslyn to run the generator, which does all expensive work, just to determine that is has nothing to generate.

Risk

I would consider this low. This is a non-trivial amount of code going in. However, it has been extensively tested in roslyn, with a large test suite, and has been validated for the 7.0 sdk. That said, incremental source generation is itself complex (one of the reasons that the perf issue arose in teh first place), so this is definitely more than a trivial fix.

Link the PR to the original issue and to the PR to main.

Issue PR:
#70754

Main PRs:
#71653
#71651
#70911

This was multiple PRs as i separated out the initial polyfill, and then did a single PR per generator to update.

Note any packaging impact. (If the changes touch code in a package that we ship out of band (OOB) then we will need to take care to ensure those packages ship.)

Yes. This impacts the Microsoft.Extensions.Logging.Abstractions NuGet package as well as the System.Text.Json NuGet package. Those two ship source generators that are being edited here so we will need to service those two.

Note any ref pack impact. (Touches files contained in Microsoft.NETCore.App.Ref, Microsoft.AspNetCore.App.Ref, or Microsoft.WindowsDesktop.App.Ref).

Yes. This impacts the Microsoft.NETCore.App.Ref since the Json Source generator ships in the ref pack as well, and it is being serviced by this change.

Author: CyrusNajmabadi
Assignees: CyrusNajmabadi
Labels:

Servicing-consider, area-Extensions-Logging

Milestone: 6.0.x

Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging generator changes LGTM

@rbhanda rbhanda added Servicing-approved Approved for servicing release and removed Servicing-consider Issue for next servicing release review labels Jul 19, 2022
@rbhanda rbhanda modified the milestones: 6.0.x, 6.0.9 Jul 19, 2022
@rbhanda
Copy link
Contributor

rbhanda commented Jul 19, 2022

Merge depends on 17.4 Preview 1 testing.

@ghost
Copy link

ghost commented Aug 2, 2022

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

Backport of: #71653 and #71651.

Bulk of code added in #70911. However that PR also updated the RegexGenerator (which does not exist in 6.0 and thus does not need to be serviced).

Note: this is a polyfill of an API that roslyn has added. Once SDK can move to a version of roslyn containing the api, it can reference that API directly and remove the 1000 lines of added code for the polyfill.

Servicing Template:

Description:

Primary goal here is to use a more efficient incrementally built up system for determining what classes should be examined to see if we need the source generators to run. Previously, all types in a solution would be examined (costly in CPU and memory), now only types with attributes on it (that we can show could actually bind to the desired attribute) are checked. As in practice this is a tiny subset of actual types in a solution (often zero) this vastly decreases the amount of work we needed to be done.

--

Port was very clean, requiring only one deviating line between it and the code that exists in the 7.0 line.

Customer Impact:

Improved performance of 'in box' generators when in visual studio. The existing two generators cause lots of CPU and memory churn when processing. This is because each of them effectively walks all files/types looking for potential matches. This forces continual reparsing of files, and then lots of semantic binding on potential matches. For large projects (consider thousands of files being normal), this is an enormous amount of work that can use tons of CPU and memory. In Roslyn this can be particularly bad, leading to our server getting bogged down and slowing down all language services.

The new approach uses the incremental generator engine to incrementally build up a list of 'actually viable classes' to examine. The intuition generally being: if you only care about types with "JsonSerializable" on them, then only check types that have that attribute on it syntactically. This is slightly more involved than just a trivial name check because things like aliases/global-aliases mean that someone could go global using Foo = System.Text.Json.JsonSerializable; in one file and then [Foo] class MyClass in another. However, with this, we can narrow down to only the set of types that are actually relevant, not all types. Furthermore, in projects that reference the sdk but do not use the generator at all, there will be zero potential matches, and so no semantic work has to happen at all.

Regression?

Yes. This was a perf (but not functionality) regression for anyone with a large solution and who moved to .net 6. Note taht a customer doesn't need to be using the generator to have the perf hit. Just referencing the SDK is enough as it will cause Roslyn to run the generator, which does all expensive work, just to determine that is has nothing to generate.

Risk

I would consider this low. This is a non-trivial amount of code going in. However, it has been extensively tested in roslyn, with a large test suite, and has been validated for the 7.0 sdk. That said, incremental source generation is itself complex (one of the reasons that the perf issue arose in teh first place), so this is definitely more than a trivial fix.

Link the PR to the original issue and to the PR to main.

Issue PR:
#70754

Main PRs:
#71653
#71651
#70911

This was multiple PRs as i separated out the initial polyfill, and then did a single PR per generator to update.

Note any packaging impact. (If the changes touch code in a package that we ship out of band (OOB) then we will need to take care to ensure those packages ship.)

Yes. This impacts the Microsoft.Extensions.Logging.Abstractions NuGet package as well as the System.Text.Json NuGet package. Those two ship source generators that are being edited here so we will need to service those two.

Note any ref pack impact. (Touches files contained in Microsoft.NETCore.App.Ref, Microsoft.AspNetCore.App.Ref, or Microsoft.WindowsDesktop.App.Ref).

Yes. This impacts the Microsoft.NETCore.App.Ref since the Json Source generator ships in the ref pack as well, and it is being serviced by this change.

Author: CyrusNajmabadi
Assignees: CyrusNajmabadi
Labels:

Servicing-approved, area-Meta

Milestone: 6.0.9

@tarekgh
Copy link
Member

tarekgh commented Aug 2, 2022

I labeled it Meta as it is covering multiple areas.

@carlossanlop
Copy link
Member

Merge depends on 17.4 Preview 1 testing.

@rbhanda who should take care of this so I can merge it?

@carlossanlop
Copy link
Member

Merge depends on 17.4 Preview 1 testing.

@rbhanda @joperezr @CyrusNajmabadi is this good to merge or do we still need confirmation of the testing? Today is the due date for merging backports that we intend to add to the September servicing release.

@carlossanlop
Copy link
Member

Tactics approved.
Milestone applied.
CI passed.
OOB package authoring changes included.
Signed off by area owners.
From a chat conversation with @joperezr, we didn't hear about any testing issues, so we're good.
Ready to merge! :shipit:

@carlossanlop carlossanlop merged commit 5a0d081 into release/6.0 Aug 15, 2022
@carlossanlop carlossanlop deleted the portPolyfill branch August 15, 2022 23:37
@ghost ghost locked as resolved and limited conversation to collaborators Sep 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Meta Servicing-approved Approved for servicing release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants