-
-
Notifications
You must be signed in to change notification settings - Fork 535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using pytest for integration tests. #4125
Using pytest for integration tests. #4125
Conversation
Signed-off-by: Pradyot Ranjan <99216956+pradyotRanjan@users.noreply.github.com>
I merged |
This is looking very good for local development purposes, and would have been even better if we had larger GitHub runners. Here are the results from a few local runs from my macOS machine with 11 cores: Running the integration tests via serial execution
============================== 344 passed, 1 skipped in 563.97s (0:09:23) ============================== Running the integration tests via parallel execution
============================== 344 passed, 1 skipped in 100.53s (0:01:40) ============================== From |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, thanks, @prady0t! As discussed, cleaning up the run-tests.py
file will be great to look at in the coming weeks, there are a lot of if-else conditions in there. I did test out run-tests.py --unit
and run-tests.py --integration
along with their nox
equivalents, and all of them are collecting the expected tests.
Signed-off-by: Pradyot Ranjan <99216956+pradyotRanjan@users.noreply.github.com>
…testing-integration-tests
Are we planning to eventually completely remove |
Yes, it looks like there is a bunch of legacy, verbose code that is better replaced by keeping all of the testing commands/infra inside |
Signed-off-by: Pradyot Ranjan <99216956+pradyotRanjan@users.noreply.github.com> Co-authored-by: Pradyot Ranjan <99216956+pradyotRanjan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Agriya Khetarpal <74401230+agriyakhetarpal@users.noreply.github.com>
Description
Running integration tests with pytest. Almost 50% of the integration tests were failing due to multiple tests accessing
test_model.json
file leading to race conditions and file corruption. We have useduuid
to generate unique filename for every test. With this approach we can run all integration tests in parallel. Overall time to run integration tests is approx. 7-8 min. nowAddresses #3617
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.
Key checklist:
$ pre-commit run
(or$ nox -s pre-commit
) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)$ python run-tests.py --all
(or$ nox -s tests
)$ python run-tests.py --doctest
(or$ nox -s doctests
)You can run integration tests, unit tests, and doctests together at once, using
$ python run-tests.py --quick
(or$ nox -s quick
).Further checks: