Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --exact flag #2527

Merged
merged 17 commits into from
Jun 19, 2023
Merged

Add --exact flag #2527

merged 17 commits into from
Jun 19, 2023

Conversation

jaisnan
Copy link
Contributor

@jaisnan jaisnan commented Jun 14, 2023

Description of changes:

When the user calls kani with a command cargo kani --harness module::check_blah --exact and they want to specify which harness to match, using --exact, they can prevent multiple harnesses from being run. The user will need to specify the harness name by using the fully qualified name as well.

Resolved issues:

Resolves #2285

Call-outs:

If the user adds --exact and they want to run kani on multiple harnesses, they will need to provide the specific name for every harness.

Testing:

  • How is this change tested? Unit and regression change

  • Is this a refactor change? no

Checklist

  • Each commit message has a non-empty body, explaining why the change was made
  • Methods or procedures are documented
  • Regression or unit tests are included, or existing tests cover the modified code
  • My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

@jaisnan jaisnan requested a review from a team as a code owner June 14, 2023 20:22
Copy link
Contributor

@zhassan-aws zhassan-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jaisnan! I was expecting to see at least one test with two or more harnesses where the name of one harness is a substring of the other, and --exact --harness is used for selecting the one that is a substring while making sure the other one is not selected.

tests/ui/exact-harness/some_matching_harnesses/expected Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good. Just a few minor comments.

I do wonder if we should fail in the case where an exact match was not found. I don't think rust tests fail, but they also don't fail when a filter has no match.

@adpaco-aws might have an opinion about this one. @jaisnan do you have any preference?

kani-driver/src/args/mod.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
tests/ui/exact-harness/some_matching_harnesses/subset.rs Outdated Show resolved Hide resolved
tests/ui/exact-harness/incomplete-harness-name/expected Outdated Show resolved Hide resolved
@celinval
Copy link
Contributor

Thanks @jaisnan! I was expecting to see at least one test with two or more harnesses where the name of one harness is a substring of the other, and --exact --harness is used for selecting the one that is a substring while making sure the other one is not selected.

I think the test tests/ui/exact-harness/some_matching_harnesses/subset.rs was design to test that, but I don't think it is quite testing it yet.

@adpaco-aws
Copy link
Contributor

I do wonder if we should fail in the case where an exact match was not found. I don't think rust tests fail, but they also don't fail when a filter has no match.

@adpaco-aws might have an opinion about this one.

Yes, I do. We should fail in that case. In addition to "catch" mistakes from users, it'll also be helpful to detect issues with target selection on the VSCode extension.

celinval
celinval previously approved these changes Jun 15, 2023
Copy link
Contributor

@celinval celinval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider adding the number of ignored harnesses to the summary. But that's out of the scope of this change. Thanks Jai

@celinval celinval dismissed their stale review June 15, 2023 22:57

Actually, we're still missing the failure when an exact match was not found.

@jaisnan
Copy link
Contributor Author

jaisnan commented Jun 16, 2023

@celinval I've added the error when an exact match is not found.

kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
kani-driver/src/metadata.rs Outdated Show resolved Hide resolved
tests/ui/exact-harness/fail_on_missing/subset.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@zhassan-aws zhassan-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jaisnan!

@jaisnan jaisnan merged commit 3b5c7c7 into model-checking:main Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kani doesn't have an --exact flag to call for singular harnesses
4 participants