Better control of test-execution order

[lukaslueg]

Cargo currently does not take too much care about what order tests are executed. cargo_test::compile_tests() places them in order of (PackageID, TargetKind, Name). We could do better than that by ordering test execution by

1. If the test has failed before
2. If the test has been executed at all
3. (PackageID, TargetKind)
4. The time it took the last time to execute this test, in descending order
5. Name

This has some benefits:

• Failing tests are executed first, without the need to stop cargo watch and filter the test we just broke.
• Tests which have not been executed at all (due to fail-fast) are executed as soon as possible, so we get a least one reading
• Slow tests are scheduled first, so their execution overlaps as long as possible, avoiding having just one or two slow tests be scheduled at the end and consuming just one cpu.

Notice that if the testsuite completes successfully, the ordering almost returns to current behaviour, with only latency-scheduling taking precedence over Name.

To do this, we'd need to persists some information to disk. We could do this by writing a "marker"-file somewhere in /target for every test (consider "target/testruns/test-hash(...)") where the files' ctime is the time the test were last started and the files' mtime is the time the tests were last completed successfully. We can reconstruct the above ordering from that.
Alternatively we serialize a single struct to disk.
Also notice that the ordering we derive from this information may be wrong or outdated (e.g. by changing test-bodies), yet this does not cause the testsuite to execute incorrectly as a whole and Is Only Wrong Once™

See also #10673

[lukaslueg]

Oh and I can probably implement this if so desired.

[epage]

This would likely require changing the relationship between cargo and libtest which I plan to do for #5609.

With that said, I would recommend experimenting with the policy for this in cargo-nextest before we evaluate such a policy for cargo.

[sanmai-NL]

Another value add: sometimes a user is involved in software based (scientific) experiments. While automated tests should not be dependent on their order of execution, ideally, sometimes you do see a certain order dependence with a certain phenomenon you investigate. For example, related to performance, efficiency, energy efficiency, or security. Being able to control execution order would be valuable in such use cases as well. In this context I now use cargo-nextest.