Build failed: dotnet-arcade-services-weekly/main #2023022701

[dotnet-eng-status]

Build #2023022701 failed

❌ : internal / dotnet-arcade-services-weekly failed

Summary

Finished - Mon, 27 Feb 2023 12:19:09 GMT
Duration - 6 minutes
Requested for - Microsoft.VisualStudio.Services.TFS
Reason - schedule

Details

SynchronizeSecrets

• ❌ - [Log] - Key Vault Secret 'deployment-staging-api-key' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ⚠️ - [Log] - Extra secret 'deployment-staging-api-key' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dnc-eng-app-insights-api-key' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-eng-app-insights-api-key' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-eng-grafana-staging-westus2-cloudapp-azure' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-eng-grafana-westus2-cloudapp-azure' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-grafana-github-oauth-app-secret' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-grafana-production-github-oauth-app-secret' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotnet-grafana-staging-github-oauth-app-secret' consider deleting it.

• ⚠️ - [Log] - Extra secret 'dotneteng-status-app-insights-api-key' consider deleting it.

• ⚠️ - [Log] - Extra secret 'grafana-admin-api-key-hot' consider deleting it.

• ❌ - [Log] - Key Vault Secret 'dnc-eng-app-insights-api-key' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'deployment-table-sas-uri-FAKE' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'app-insights-instrumentation-key' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'app-insights-instrumentation-key' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'BotAccount-dotnet-gh-app-bot-otp' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'dnceng-code-rw' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'azure-appconfiguration-connection-string' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

• ❌ - [Log] - Key Vault Secret 'build-asset-registry-sql-connection-string-read' is missing next-rotation-on tag, using the end of time as value. Please force a rotation or manually set this value.

Changes

• d24bed6e - Alex Perovich - Produce an error when a secret does not have next-rotation-on ([Maestro/Darc] Add " #2225)
• f798c45c - Djuradj Kurepa - Dkurepa/fix kusto.ingest bump (source build prebuilts #2228)

[MattGal]

If we don't address this soon, the next time an arcade-services secret expires it will fall on the floor.

[premun]

Additionally, we should stop running unit tests in the rolling build as it can fail every now and then: https://dev.azure.com/dnceng/internal/_build/results?buildId=2124757&view=logs&j=a5c6a902-5556-5a70-4b67-7f49bfe8ebe3&t=30522ad2-fb1e-5545-250b-00b27148e543&s=6884a131-87da-5381-61f3-d7acc3b91d76

[tkapin]

Additionally, we should stop running unit tests in the rolling build as it can fail every now and then: https://dev.azure.com/dnceng/internal/_build/results?buildId=2124757&view=logs&j=a5c6a902-5556-5a70-4b67-7f49bfe8ebe3&t=30522ad2-fb1e-5545-250b-00b27148e543&s=6884a131-87da-5381-61f3-d7acc3b91d76

I'd like us to understand what's the reason of the tests being flaky before taking any decision on disabling them.

[premun]

@tkapin there are no test of ours actually failing. The code coverage tool sometimes crashes when trying to read the code coverage run config.

For other repositories, we never run tests again in the internal builds and these are adding 10+ minutes to an already long build.

[ilyas1974]

While evaluating what is run is a great idea and should be done, I want to make sure we prioritize updating the secret here and not block rollouts next week. @premun, can you open a separate issue for your recommendation on modifying what we are running in this pipeline.

[premun]

Done: dotnet/arcade-services#2324

[andriipatsula]

@ilyas1974 , we have a conversation in the General channel in Teams. In order to unblock the pipeline, I prepared a PR: dotnet/arcade-services#2234 that lowers the severity of log record (introduced this week in the dotnet/arcade-services@d24bed6 commit). We can go with it or we can choose long path by resolving 130 secrets and adding missing tags.

[MattGal]

@ilyas1974 , we have a conversation in the General channel in Teams. In order to unblock the pipeline, I prepared a PR: dotnet/arcade-services#2234 that lowers the severity of log record (introduced this week in the dotnet/arcade-services@d24bed6 commit). We can go with it or we can choose long path by resolving 130 secrets and adding missing tags.

My opinion no longer matters here, but I just want to register opposition to doing this. This error specifically needs to exist because we have had secrets fail to get cycled and affect production systems multiple times for the reason the error is trying to block. Downgrading it to a warning is downgrading it to being ignored like all the other warnings.

[garath]

@dkurepa I think you took care of this with dotnet/arcade-services#2238, yes?

[premun]

Yes, I'd close this (even though the fix was not verified yet as we got hit with some new CG errors, but i'd close this anyway, worst case a new one pops up)