Helix-machines has flaky test execution

[riarenas]

• This issue is blocking
• This issue is causing unreasonable pain
  PRs to the repo are very painful because even small changes need a lot of attempts to get past both flaky problems.

Helix-machines build attempts are regularly failing with one of two issues:

1. dotnet.exe returning exit code 1 without any info during the build step, just after restoring projects: https://dnceng.visualstudio.com/internal/_build/results?buildId=2161729&view=logs&j=3dc8fd7e-4368-5a92-293e-d53cefc8c4b3&s=6884a131-87da-5381-61f3-d7acc3b91d76&t=7510ef29-db76-5a4c-4d87-8cd81d6b33b8&l=83
2. dotnet returning failure exit codes during test execution: https://dnceng.visualstudio.com/internal/_build/results?buildId=2160507&view=logs&j=3dc8fd7e-4368-5a92-293e-d53cefc8c4b3&t=baf0d620-26ff-5880-4e4c-71df819d0940

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[riarenas]

Here is a PR build that saw these issues throughout 4 attempts: https://dnceng.visualstudio.com/internal/_build/results?buildId=2163280

Attempts 1, 2 and 4 all saw issue the problem in the build step, while attempt 3 saw the problem with the test step. Attempt 5 completed the build job successfully.

[riarenas]

The test issues are exactly the same as the ones reported in dotnet/arcade-services#2324 for arcade-services.

Some things that we can try here:

• Updating the dotnet sdk we use in the repo
• Updating the version of vstest we use in the repo (I think this might just come from the sdk in helix-machines case)
• Updating the version of Nuget, in case the problems are related to the central package management features.

[dougbu]

see also

• NuGet restore fails with The repository primary signature validity period has expired arcade#13070 which corresponds to (1) above i.e., we do have more information but the real errors are logged in white,
• BuildImages errors about C:\Program Files (x86)\Windows Kits\10\Debuggers from windows-public-debuggers.ps1 and similar errors about C:\Program Files\dummyName from windows-docker.ps1,
• BuildImages "Custom Script Extension failed to deploy" warnings that may cause failures of that job, and
• DeployQueues flakiness, especially for Mariner images

[riarenas]

I think I would like this issue to only be about the build stage in helix-machines. Issues related to images seem like they should be dealt with differently as they have to do with the images.

Any errors in deployqueues and createCustomImage are more involved than just building the applications that execute those stages.

Can you explain a bit about how you see dotnet/arcade#13018 being related? that seems exclusively a publishing infrastructure bug and the error doesn't seem to match what I'm seeing here.

[dougbu]

My bad. I meant dotnet/arcade#13070 but hash-autocomplete failed me. Editing…

[riarenas]

That makes some more sense. I still think this is something different, because every time we see the signature issue, we get that feedback. It's totally possible that both issues would be solved by updating to a newer NuGet though?

[dougbu]

That makes some more sense. I still think this is something different, because every time we see the signature issue, we get that feedback. It's totally possible that both issues would be solved by updating to a newer NuGet though?

Why do you think the two restore problems are different❔ The link you provided shows error from dotnet/arcade#13070 in nice clear (white) letters.

On updating NuGet, I have a branch that updates the Arcade SDK and therefore the .NET SDK. that might do it.

Question is how to move forward once 'main' is green❔ Let's talk ordering in the next little while.

[riarenas]

Ooh, that makes even more sense. I definitely missed that in my builds. I think the failures during test execution are the ones that are unaccounted for then.

[riarenas]

As a summary:

• The errors during the build step are all NuGet restore fails with The repository primary signature validity period has expired arcade#13070.
• The errors during the test steps such as: https://dnceng.visualstudio.com/internal/_build/results?buildId=2163280&view=logs&j=7871e8ac-1348-5c5a-3691-0fa1e72e8e2e&t=0197cea2-2421-5a31-ab13-698b2418f1de&s=b49e7e65-b64b-557c-018e-3fd79c15ded6 need to be investigated.

[dougbu]

I looked at the test failure a bit. Unfortunately, the dotnet-helix-machines-ci Tests analytics lack tests run in the Test C# job. I'm sure we've seen similar errors since @riarenas opened this issue but can't verify that w/o searching through all failed and passed on retry runs in the pipeline.

However, the failure linked just above occurred in the Dispose() method of test/Script.Tests/Tests.cs after otherwise successfully running UnicodePathTest(). The logging shows an unexpected leftover file caused an Assert.Empty(...) call to fail.

[xUnit.net 00:00:54.09]     ScriptTests.Tests.UnicodePathTest [FAIL]
  Failed ScriptTests.Tests.UnicodePathTest [1 s]
  Error Message:
   Assert.Empty() Failure
Collection: ["D:\\a\\_work\\_temp\\.artifacts\\helix-scripts\\te"...]
  Stack Trace:
     at ScriptTests.Tests.Dispose() in D:\a\_work\1\s\test\Script.Tests\Tests.cs:line 58
   at ReflectionAbstractionExtensions.DisposeTestClass(ITest test, Object testClass, IMessageBus messageBus, ExecutionTimer timer, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Extensions\ReflectionAbstractionExtensions.cs:line 79
...
 =========== WORKITEM CONSOLE OUTPUT ==============
 Console log: 'SomeName' from job 00000000-0000-0000-0000-00000000000f workitem 00000000-0000-0000-0000-000000000010 (testqueue) executed on machine 13faa7cac000000 running Windows-10-10.0.20348-SP0
 ['SomeName' END OF WORK ITEM LOG: Command exited with 0]
 New files found in %TEMP% directory:
D:\a\_work\_temp\.artifacts\helix-scripts\temp\lbozsokg

@dotnet/dnceng anyone familiar enough w/ that test class or how helix-scripts/ uses $env:TEMP to guess what is likely to write out a seemingly-random 8 character file or directory❔ (May be worth adding more information to the output e.g., the contents of the directory or at least whether it's a file or a directory.)

[markwilkie]

Added to Helix Epic

[dougbu]

unassigning myself because I don't understand the relevant tests and am not sure whether they'll need to change in other ways for the Helix epic.