Support flags for cargo nextest?

[jszwedko]

Hi all,

I was trying out cargo nextest with https://github.com/vectordotdev/vector . I'd like to be able to run the criterion benchmarks as tests too, but ran into issues due to expectations that cargo nextest has of the test harness to support the flags mentioned on https://nexte.st/book/custom-test-harnesses.html similar to libtest. Is this something you'd be open to a PR for?

Relevant:

• Support for running criterion benches as tests? nextest-rs/nextest#96
• Collaborate with libtest-mimic? nextest-rs/nextest#38

[sunshowers]

If you base criterion on libtest-mimic 0.4.0 or above, this will automatically work FYI :)

[bheisler]

Hello! Thanks for the suggestion.

I guess my feeling is that Criterion benchmarks aren't tests, so it doesn't make sense to run them with your tests. For one thing, benchmarks are automatically compiled under a different optimization profile from tests, and mixing optimized and unoptimized benchmark runs would produce confusing results. Perhaps I'm misunderstanding the goal here.

However, Criterion-rs does have a test mode, where running the benchmarks with --test executes the benchmark once without measuring it or computing statistics. That could be useful to support in a test runner like nextest.

On that basis, I would be open to a pull request adding some additional CLI options that switch it into testing mode and substitute in a nextest-compatible output formatter when it detects nextest's extra options. The relevant logic is in src/lib.rs, specifically the configure_from_args function.

Since Criterion-rs isn't really trying to be a test harness, I believe that using libtest-mimic wouldn't be appropriate. And in addition to that I think it would require more rework than optionally using a different CLI output formatter.

I would appreciate it if the nextest-related code is clearly marked by comments, though.

FYI: I did see another issue recently that proposed replacing the clap argument parser with something lighter-weight. A pull request to add additional options would likely conflict with that.

[sunshowers]

Thanks for the response!

However, Criterion-rs does have a test mode, where running the benchmarks with --test executes the benchmark once without measuring it or computing statistics. That could be useful to support in a test runner like nextest.

Yes, that's been the mode where there's been user interest to run nextest against.

Note that nextest doesn't need any options that aren't already supported by libtest. So it might just be a matter of adding a couple of flags to criterion to make it more compatible with libtest.