-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revisit tests #2811
Comments
Interesting case of tests on the line between integration and end-to-end: #2903 These tests are e2e according to the definition above, because they initialize a model and do inference on it. However, the nodes are classifiers, so for the purpose of testing, mocking the models is trivial and matter of a simple if-else. So, are these tests better seen as end-to-end due to the inference step, or they should rather be mocked and used as integration tests? Note that these, in my opinion, are too high level to be considered unit tests, regardless of the presence of the mock or their speed. They are testing most of the node at once and imho this is too wide of a scope. |
Add the outcome of #2811 to the developers docs Ideally, newly added tests will follow those requirements while we progressively adapt the existing tests to the new model.
* Update CONTRIBUTING.md Add the outcome of #2811 to the developers docs Ideally, newly added tests will follow those requirements while we progressively adapt the existing tests to the new model. * address review comments
We decided to split this epic and continue the work in chunks, see the updated issue description. Further conversations will happen in their respective epic. With the newly reduced scope, I'm going to call this one done and close. |
Problem
At the moment there is some confusion about how test functions are categorized and it's not easy to distinguish between unit and integration tests. Fixing this problem would improve the development experience by making the tests fail fast if there is an issue and by making it more obvious what's the issue.
Proposal
We formally define three scopes for tests in Haystack with different requirements and purposes; newly added tests will follow those requirements while we progressively adapt the existing tests to the new model.
The three scopes are defined as follows:
draft
andready
PRs, automated through pytestready
PRs, automated through pytestAction plan
The text was updated successfully, but these errors were encountered: