Investigate helix-machines failures.

[riarenas]

• This issue is blocking
• This issue is causing unreasonable pain

https://dnceng.visualstudio.com/internal/_build/results?buildId=2160719&view=results

latest build was a mess. A lot of images failed. Investigate and try to figure out.

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[riarenas]

We were finally able to work through issues (mostly by reverting and ignoring errors) and got a successful build in https://dnceng.visualstudio.com/internal/_build/results?buildId=2163409&view=results

As a summary of what is still pending:
Issues directly blocking the repo

• Resolve the Component Governance issue in the dotnet-helix-machines repository arcade#13186: We are out of SLA in component governance for the python cryptography package. We need to thoroughly test this change in the older OSes.

• windows.vs2017.amd64(.open) image generation timing out during git installations arcade#13231: windows.vs2017.amd64(.open) time out during the windows-git artifact installation. We can try updating the version of git we install to see if that resolves things.

issues that are not blocking, but cause unreasonable pain as we have to retry builds a lot:

• Helix-machines has flaky test execution #1320: dotnet crashes during the test leg, not sure why

• NuGet restore fails with The repository primary signature validity period has expired arcade#13070: this causes failures during restore, we saw about 10 of these just yesterday in helix-machines builds

These are issues for improvements to processes or minor annoyances:

• Improve the FORCE_QUEUE process in Helix-machines arcade#13212: Improve the FORCE_QUEUE feature for PR validation. One of the root causes for our python dependencies problems in older OSes was that the PRs didn't test the images that then failed to generate post-merge

• Decrease the number of CG scans that helix-machines builds perform arcade#13248: We run the CG scans in too many stages during a single build. The latest main build has over 20 warnings in different legs that all refer to the same CG alert.

Additionally, during the latest main build that built all the windows images, we saw some flakiness in some artifacts like the windows-debuggers where we fail to find some files that the artifacts need. Later attempts were successful so I'm hesitant to open issues specific about these as they smell like antivirus kicking in at bad times. We can investigate more if we think it's worth it

[dougbu]

I suggest closing this b/c all of the work seems to be covered in other issues. If some of the issues remain open in dotnet/arcade, suggest moving them to dotnet/dnceng. I did this for #1320.

[dougbu]

Oh, none of the other issues referenced above remain open. #1320 is all we have left.

[ilyas1974]

As there are just other issues being referenced here, I am going to close this issue.