DefaultDiagnosticUpdateSourceTests contains flaky tests

[333fred]

The following are known flaky tests within this class:

• TestDefaultDiagnosticProviderSemantic
• TestDefaultDiagnosticProviderRemove
• TestDefaultDiagnosticProviderAll

Details for TestDefaultDiagnosticProviderSemantic

This test has failed 15 times in the last 14 days. Failure message:

Expected:
CS0246
Actual:
CS0246,
CS0246
Differences:
    CS0246,
++> CS0246

Expected: True
Actual:   False

Some examples of failures:
https://dev.azure.com/dnceng/public/_build/results?buildId=724700&view=ms.vss-test-web.build-test-results-tab
https://dev.azure.com/dnceng/public/_build/results?buildId=721954&view=ms.vss-test-web.build-test-results-tab
https://dev.azure.com/dnceng/public/_build/results?buildId=718388&view=ms.vss-test-web.build-test-results-tab

One possibly notable detail: failures seem to take ~2.2 seconds for the test to run, successes take under .5 seconds for the most part.

Details for TestDefaultDiagnosticProviderRemove

Builds

Build	Pull Request	Test Failure Count
#739651	#46176	1
#739718	#46179	1
#739874	#46183	1
#739896	#45800	1
#740061	#46194	1
#740083	#46138	1
#740155	#45806	1
#740370	Rolling	1
#740378	Rolling	1
#740397	Rolling	1
#740404	#46189	1
#740484	#44447	1
#740543	#46199	1
#740831	Rolling	1
#741237	Rolling	1
#741316	Rolling	1
#741604	#46219	1
#741683	#46038	1

Configurations

• Windows Desktop Debug Test32
• Windows Desktop Debug Test64
• Windows Desktop Release Test32
• Windows Desktop Release Test64
• Windows Desktop Spanish

Details for TestDefaultDiagnosticProviderAll

roslyn/src/VisualStudio/Core/Test/Diagnostics/DefaultDiagnosticUpdateSourceTests.vb

Line 143 in 1386d0f

Public Async Function TestDefaultDiagnosticProviderAll() As Task

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=749582&view=ms.vss-test-web.build-test-results-tab&runId=23240534&resultId=202015&paneView=debug

Xunit log:

Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.TestDefaultDiagnosticProviderAll [STARTING]
      
      Expected:
      CS1002
      CS0246
      Actual:
      CS1002
      CS1002
      CS0246
      CS0246
      Differences:
          CS1002
      ++> CS1002
          CS0246
      ++> CS0246
      
      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/Test/Utilities/Portable/Assert/AssertEx.cs(442,0): at Roslyn.Test.Utilities.AssertEx.SetEqual[T](IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector)
        /_/src/Test/Utilities/Portable/Assert/AssertEx.cs(425,0): at Roslyn.Test.Utilities.AssertEx.SetEqual(IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector)
        /_/src/VisualStudio/Core/Test/Diagnostics/DefaultDiagnosticUpdateSourceTests.vb(177,0): at Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.VB$StateMachine_4_TestDefaultDiagnosticProviderAll.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.TestDefaultDiagnosticProviderAll [FINISHED] Time: 1.6519875s

[mavasani]

I'll take a look.

[mavasani]

Assigning to @sharwell based on the investigations in #46255

[sharwell]

@mavasani This doesn't appear to be the same issue as #46255. ☹️

[RikkiGibson]

Another failure in https://dev.azure.com/dnceng/public/_build/results?buildId=752137&view=logs&j=25fa373a-98a5-5235-2ed8-ead0b7a28f80&t=2f2a122e-a512-5643-f332-5c2c88cb7bf1&s=d654deb9-056d-50a2-1717-90c08683d50a

[sharwell]

This seems to affect this line:

roslyn/src/Compilers/CSharp/Portable/Binder/Binder_Symbols.cs

Line 2386 in 075e9c3

return diagnostics.Add(ErrorCode.ERR_SingleTypeNameNotFound, location, whereText);

It feels like parallel binding of the same identifier (optimistic initialization) is resulting in a case where one of the side effects (error reporting) is observed from both the path that wins the initialization and the one that doesn't.