Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rethink test strategy #755

Closed
mnonnenmacher opened this issue Aug 10, 2018 · 5 comments
Closed

Rethink test strategy #755

mnonnenmacher opened this issue Aug 10, 2018 · 5 comments

Comments

@mnonnenmacher
Copy link
Member

Currently most functional tests work by comparing analyzer or scan results with a YAML file that contains the expected serialized result.
The problem with this approach is, that comparing the complete file often compares much more than is required to test a single feature. As a result simple refactorings can break a lot of tests which do not test the refactored code, and it is required to change a lot of YAML test files to fix this, which makes refactorings very expensive.
A way to fix this would be to reduce the amount of YAML based tests and to make sure that a test for a specific feature only tests this feature and nothing else. This would make it necessary to write a lot more code in the tests themselves, but with the gain that refactorings become cheaper and issues with related tests can often be found at compile time, not only when executing tests.
@mnonnenmacher mnonnenmacher mentioned this issue Sep 18, 2018
Merged
@sschuberth sschuberth mentioned this issue Oct 9, 2018
Merged
@sschuberth sschuberth mentioned this issue Oct 30, 2018
Merged
sschuberth added a commit that referenced this issue Feb 21, 2019
@sschuberth
Travis: Add a matrix build for OS X
7b49df7 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Feb 21, 2019
@sschuberth
Travis: Add a matrix build for OS X
e47b6c2 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Mar 27, 2019
@sschuberth
Travis: Add a matrix build for OS X
5408f06 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Mar 28, 2019
@sschuberth
Travis: Add a matrix build for OS X
b7b3726 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Apr 1, 2019
@sschuberth
Travis: Add a matrix build for OS X
43916f7 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Apr 3, 2019
@sschuberth
Travis: Add a matrix build for OS X
ddbc3fb 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Apr 3, 2019
@sschuberth
Travis: Add a matrix build for OS X
87be76a 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue May 10, 2019
@sschuberth
Travis: Add a matrix build for OS X
e4b8035 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Jul 31, 2019
@sschuberth
Travis: Add a matrix build for OS X
63fad55 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@here.com>
sschuberth added a commit that referenced this issue Jul 31, 2019
@sschuberth
Travis: Add a matrix build for OS X
61bfed8 
To guard against OS X specific issues like #1233. Do not use CircleCI as
another service here as they only offer "1 macOS container and up to 500
build minutes per month" which is far too little for our test suite
which currently takes > 20 minutes per PR (also see issue #755).

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch-si.com>
sschuberth added a commit that referenced this issue Dec 9, 2019
@sschuberth
downloader: Move VCS tests to functional tests
0eafb16 
These are not really (unit) tests on small units of code, but tests that
perform rather length operations on VCS working trees with the help of
external CLI tools. As a consequence, reclassify them as functional
tests, which fits better, but probably still needs some clean-up in the
context of #755.

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch-si.com>
sschuberth added a commit that referenced this issue Dec 9, 2019
@sschuberth
downloader: Move VCS tests to functional tests
6b4c532 
These are not really (unit) tests on small units of code, but tests that
perform rather lengthy operations on VCS working trees with the help of
external CLI tools. As a consequence, reclassify them as functional
tests, which fits better, but probably still needs some clean-up in the
context of #755.

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch-si.com>
sschuberth added a commit that referenced this issue Dec 10, 2019
@sschuberth
downloader: Move VCS tests to functional tests
498b822 
These are not really (unit) tests on small units of code, but tests that
perform rather lengthy operations on VCS working trees with the help of
external CLI tools. As a consequence, reclassify them as functional
tests, which fits better, but probably still needs some clean-up in the
context of #755.

Signed-off-by: Sebastian Schuberth <sebastian.schuberth@bosch-si.com>
@sschuberth
Copy link
Member

Also see #2450 (comment).

@sschuberth sschuberth mentioned this issue Feb 2, 2021
Merged
@mnonnenmacher
Copy link
Member Author

mnonnenmacher commented Mar 18, 2021 •
edited
Loading

Another issue with the YAML based tests is that it is difficult to fix them when the model is changed. Issues with expected result files are discovered only when executing tests, and manually fixing them can be a lot of effort.
One approach to improve this situation is to create the expected results in Kotlin code, this way issues with model refactorings would already be detected at compile time. This is similar to the test data that is already used for some other tests, see:

To amount of code required to create this test data could be reduced by creating a simple DSL to create ORT results.

@nnobelis
Copy link
Member

One can use DSL builder in Kotlin to build expected results :
https://kotlinlang.org/docs/type-safe-builders.html

@sschuberth sschuberth mentioned this issue Jun 4, 2021
Merged
@sschuberth
Copy link
Member

One can use DSL builder in Kotlin to build expected results : https://kotlinlang.org/docs/type-safe-builders.html

Indeed, and a start was committed with 0f26841.

@sschuberth
Copy link
Member

Closed as part of backlog grooming. Feel free to comment if you would like to contribute to this.

@mnonnenmacher mnonnenmacher closed this as not planned Won't fix, can't repro, duplicate, stale Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sschuberth @mnonnenmacher @nnobelis