Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix incrementalism when suppressed: #9717

Merged
merged 5 commits into from
Jan 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.AspNetCore.Razor;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Diagnostics;
Expand Down Expand Up @@ -52,11 +53,52 @@ internal static IncrementalValueProvider<TSource> ReportDiagnostics<TSource>(thi
return source.Select((pair, ct) => pair.Item1!);
}

internal static IncrementalValuesProvider<T> EmptyWhen<T>(this IncrementalValuesProvider<T> provider, IncrementalValueProvider<bool> checkProvider, bool check)
internal static IncrementalValuesProvider<T> EmptyOrCachedWhen<T>(this IncrementalValuesProvider<T> provider, IncrementalValueProvider<bool> checkProvider, bool check)
{
return provider.Combine(checkProvider)
.Where(pair => pair.Right != check)
// This code is a little hard to understand:
// Basically you can think about this provider having two states: 'on' and 'off'.
// When the checkProvider equals the check value, the provider is 'on'
// When in the 'on' state, data flows through it as usual.
// When in the 'off' state you get either: an empty provider, if it has never been 'on' before, or the last 'on' data, but all cached.

// First we create a check provider that 'latches'. That is, once it has been flipped into the 'on' state, it never goes back to 'off'
var latchedCheckProvider = checkProvider.WithLambdaComparer((old, @new) => IsOn(old) || old == @new);

// Next we filter on the latched provider. When the provider is off we return an empty array, when it's on we allow data to flow through.
var dataProvider = provider.Combine(latchedCheckProvider)
.Where(pair => IsOn(pair.Right))
.Select((pair, _) => pair.Left);

// Now, we compare against the real value of the check provider. If the real provider is 'on' we allow the data through. When the provider is
// 'off' we set all the data to cached. This allows the caches to remain full, but still disable any further downstream processing.
var realProvider = dataProvider.Combine(checkProvider)

// We have to group and ungroup the data before comparing to ensure that we correctly handle added and removed cases which would otherwise
// not get compared and would be processed downstream
.Collect()

// When the real value is 'on', always say the data is modified. When the value is 'off' say it's cached
.WithLambdaComparer((old, @new) => !IsOn(@new.FirstOrDefault().Right))

// When 'on' the data will be re-evaluated item-wise here, ensuring only things that have actually changed will be marked as such.
// When 'off' the previous data was cached so nothing downstream runs.
.SelectMany((arr, _) => arr)
.Select((pair, _) => pair.Left);

return realProvider;

bool IsOn(bool value) => value != check;
}

/// <summary>
/// Ensures that <paramref name="input"/> reports as up to date if <paramref name="suppressionCheck"/> returns <see langword="true"/>.
/// </summary>
internal static IncrementalValueProvider<(T, bool)> SuppressIfNeeded<T>(this IncrementalValueProvider<T> input, IncrementalValueProvider<bool> suppressionCheck)
{
return input
.Combine(suppressionCheck)
// when the suppression check is true, we always say its up to date. Otherwise we perform the default comparison on the item itself.
.WithLambdaComparer((old, @new) => @new.Right || EqualityComparer<T>.Default.Equals(old.Left, @new.Left));
}

internal static IncrementalValueProvider<bool> CheckGlobalFlagSet(this IncrementalValueProvider<AnalyzerConfigOptionsProvider> optionsProvider, string flagName)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ public void Initialize(IncrementalGeneratorInitializationContext context)

// determine if we should suppress this run and filter out all the additional files and references if so
var isGeneratorSuppressed = analyzerConfigOptions.CheckGlobalFlagSet("SuppressRazorSourceGenerator");
var additionalTexts = context.AdditionalTextsProvider.EmptyWhen(isGeneratorSuppressed, true);
var metadataRefs = context.MetadataReferencesProvider.EmptyWhen(isGeneratorSuppressed, true);
var additionalTexts = context.AdditionalTextsProvider.EmptyOrCachedWhen(isGeneratorSuppressed, true);
var metadataRefs = context.MetadataReferencesProvider.EmptyOrCachedWhen(isGeneratorSuppressed, true);

var razorSourceGeneratorOptions = analyzerConfigOptions
.Combine(parseOptions)
.Combine(isGeneratorSuppressed)
.SuppressIfNeeded(isGeneratorSuppressed)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I imagine this can happen?

  1. The generator is not suppressed, the inputs (analyzerConfigOptions or parseOptions) are changing and hence we are recomputing the output (razorSourceGeneratorOptions). All is fine.
  2. The generator is suppressed.
  3. Some inputs change - but we report the output is up to date since the suppression flag is true.
  4. The suppression flag changes to false but inputs don't change. Now we still report the output is up to date - but it's not since the options changed in the previous step and we haven't recomputed the output. Seems bad.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scenario is handled (we test is here: https://github.com/dotnet/razor/pull/9717/files#diff-21bee42801ceadff30fba484ddb46162fb1af8d819b27b06e2c1ebcd612f4feeR2619)

Some of the inputs all have EmptyOrCachedWhen on them: that means when the suppression flag changes, we'll re-evaluate them, and realize that one of the items is different.

The other inputs are all used in conjunction with SuppressIfNeeded that ensures they will also be re-evaluated when that value changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I amended the test to cover the compilation changing too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test adds a document, does that change the inputs here (analyzerConfigOptions or parseOptions)? Because I'm talking specifically about SuppressIfNeeded which seems it won't re-run if the inputs change while the suppression was on - since the lambda comparer inside is returning true while suppressed, then when unsuppressed, it compares the old with the new - but the old could have changed during the suppression, hence the old would be equal to new, but could be different from the state when the suppression was off previously. For example:

  1. Suppression = off.
  2. parseOptions = A -> razorSourceGeneratorOptions = A
  3. parseOptions = B -> razorSourceGeneratorOptions = B
  4. Suppression = on.
  5. parseOptions = A -> razorSourceGeneratorOptions = B (since SuppressIfNeeded reports "up to date" due to suppression)
  6. Suppression = off -> razorSourceGeneratorOptions = B (since SuppressIfNeeded compares the previous value of parseOptions = A with the current value A hence "up to date") - seems wrong, should be changed to A.

Isn't that what could happen?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that should still work. Because we combine with the configOptions (which has to change to change suppression) ComputeRazorSourceGeneratorOptions still runs as needed and updates the options.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more to the test to cover that scenario.

Copy link
Member

@jjonescz jjonescz Jan 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks. Although it sounds like the SuppressIfNeeded works only due to the pipeline being constructed in a specific way (eg, here it works only because one of the inputs contains the suppression check). Might be worthwhile to add a warning to the doc comment of SuppressIfNeeded to make sure it's not used incorrectly in the future (if it can really behave as I describe if used on different inputs for example).

.Select(ComputeRazorSourceGeneratorOptions)
.ReportDiagnostics(context);

Expand Down Expand Up @@ -112,7 +112,7 @@ public void Initialize(IncrementalGeneratorInitializationContext context)

var tagHelpersFromCompilation = declCompilation
.Combine(razorSourceGeneratorOptions)
.Combine(isGeneratorSuppressed)
.SuppressIfNeeded(isGeneratorSuppressed)
.Select(static (pair, _) =>
{

Expand Down Expand Up @@ -199,7 +199,7 @@ public void Initialize(IncrementalGeneratorInitializationContext context)

// Currently unused. See https://github.com/dotnet/roslyn/issues/71024.
var razorHostOutputsEnabled = analyzerConfigOptions.CheckGlobalFlagSet("EnableRazorHostOutputs");
var withOptionsDesignTime = withOptions.EmptyWhen(razorHostOutputsEnabled, false);
var withOptionsDesignTime = withOptions.EmptyOrCachedWhen(razorHostOutputsEnabled, false);

var isAddComponentParameterAvailable = metadataRefs
.Where(r => r.Display is { } display && display.EndsWith("Microsoft.AspNetCore.Components.dll", StringComparison.Ordinal))
Expand Down Expand Up @@ -284,24 +284,34 @@ public void Initialize(IncrementalGeneratorInitializationContext context)
}

return string.Equals(a.csharpDocument.GeneratedCode, b.csharpDocument.GeneratedCode, StringComparison.Ordinal);
});
})
.WithTrackingName("CSharpDocuments");

context.RegisterImplementationSourceOutput(csharpDocuments, static (context, pair) =>
{
var (filePath, csharpDocument) = pair;
var csharpDocumentsWithSuppressionFlag = csharpDocuments
// Explicitly combine with the suppression state. We *do* want this to run even if we're in the latched state
.Combine(isGeneratorSuppressed)
.WithTrackingName("DocumentsWithSuppression");

// Add a generated suffix so tools, such as coverlet, consider the file to be generated
var hintName = GetIdentifierFromPath(filePath) + ".g.cs";
context.RegisterImplementationSourceOutput(csharpDocumentsWithSuppressionFlag, static (context, pair) =>
{
var ((filePath, csharpDocument), isGeneratorSuppressed) = pair;

RazorSourceGeneratorEventSource.Log.AddSyntaxTrees(hintName);
for (var i = 0; i < csharpDocument.Diagnostics.Count; i++)
// When the generator is suppressed, we may still have a lot of cached data for perf, but we don't want to actually add any of the files to the output
if (!isGeneratorSuppressed)
{
var razorDiagnostic = csharpDocument.Diagnostics[i];
var csharpDiagnostic = razorDiagnostic.AsDiagnostic();
context.ReportDiagnostic(csharpDiagnostic);
}
// Add a generated suffix so tools, such as coverlet, consider the file to be generated
var hintName = GetIdentifierFromPath(filePath) + ".g.cs";

RazorSourceGeneratorEventSource.Log.AddSyntaxTrees(hintName);
for (var i = 0; i < csharpDocument.Diagnostics.Count; i++)
{
var razorDiagnostic = csharpDocument.Diagnostics[i];
var csharpDiagnostic = razorDiagnostic.AsDiagnostic();
context.ReportDiagnostic(csharpDiagnostic);
}

context.AddSource(hintName, csharpDocument.GeneratedCode);
context.AddSource(hintName, csharpDocument.GeneratedCode);
}
});
}
}
Expand Down
Loading