Turn AutoRetry On

[sunandabalu]

No description provided.

[danmoseley]

@karelz should OK this also. Karel, this will rerun failing jobs (eg., System.Net.Http.Tests as a whole) if any fail. As configured it will only try this once. All failing logs will be shown in MC in the usual way, even if it passes on rerun. A pass on rerun will show up in the default view, albeit in green, with a curly arrow.

This should help suppress flaky failures, which we can mine from Kusto to actually do something about.

[karelz]

In general that sounds good. I am curious:

What exactly will show MC if:

1. Only 1st run fails test A?
2. 1st run fails A, 2nd run fails B?
3. 1st run fails A & B, 2nd run fails B & C?
   How will it show in build report and in test history?

Is it applied to PRs, nightly runs or both?

[sunandabalu]

The number of times a retry happens is not per test level, it is at a work item level. If a workitem has one or mote test failures, the same work item is requeued to run again.

1. Only 1st run fails test A?
   Which means in the retry attempt the test passed, test A will show up in MC as passed on retry. The Work Item will also show as passed on retry.
2. 1st run fails A, 2nd run fails B?
   If tests fails in the 1st attempt, and passes in the subsequent attempts will show up as passed on retry. Work Item will show as passed on retry if subsequent attempt was successful, otherwise will show as fail.
3. 1st run fails A & B, 2nd run fails B & C?
   Same as 2 above

Is it applied to PRs, nightly runs or both?
Applies to all Helix runs for the repo.

[karelz]

@sunandabalu are you sure [2] and [3] above will show "passed on retry" when there were failures in the retry run? I could believe that in case [2] if the comparison is per test, but in [3] it is a definitely a bug as one test consistently failed in both runs. That should definitely show as a failure.

Will I see both reruns entries in per-test history view on MC? Will I be able to click-through to the logs for each run?

[sunandabalu]

The work item will show as pass on retry only if the 2nd attempt passes ie no tests failed. In both [2] and [3] the work item will show as fail. The specific test that passed on retry will show as passedonretry.
The logs specific to each run per test will be displayed , for eg., if this test were run twice you will see two entries for the specific arch - https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180424.05/workItem/System.Net.Http.Functional.Tests/wilogs

Per Test page will show PassedOnRetry if the test passed on retry , for eg. here - https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180424.05/workItem/System.Net.Http.Functional.Tests

Per Workitem will show PassedOnRetry only if the next attempt was successful , for eg., here - https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180424.05

[sunandabalu]

@karelz can you approve the change, so I can merge the PR?

[karelz]

@sunandabalu do you have example links where it shows PassedOnRetry? The links don't seem to show that ...

[sunandabalu]

Not yet in PROD, once this PR makes it in we can get some data showing up.

[sunandabalu]

@dotnet-bot Please Test Linux-musl x64 Debug Build

[sunandabalu]

@danmosemsft , @karelz looks like I don't have access to merge to corefx, can either of you merge this?