Make sure that the PyPI distribution works

[CasperWA]

I guess the real goal of this test is to "make sure that the PyPI distribution works".

I would argue that this is tested more strongly (and with less lines of code / maintenance needs) by adding a CI test that creates the package distribution, installs it (e.g. like this) and then runs either the whole test suite or some subset of it.

Originally posted by @ltalirz in https://github.com/_render_node/MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDM2ODUxNzAwNg==/comments/review_comment

Add a GH Actions workflow to test the PyPI distribution.

[shyamd]

That can be done with tox as well, which effectively creates a virtualenv, installs it, and then runs the tests.

Do we want to ship the tests with the pypi package?

[CasperWA]

That can be done with tox as well, which effectively creates a virtualenv, installs it, and then runs the tests.

Ohh, I'll look into it, or maybe you want to?

Do we want to ship the tests with the pypi package?

At the moment we are not doing this. Only the validator. We recently pulled out all the tests from the package into its own /tests folder.

[shyamd]

ok,
do you mind if I do some package/workflow restructuring to make this all automatic?

[CasperWA]

ok,
do you mind if I do some package/workflow restructuring to make this all automatic?

No please.
Note that I have added a test for this though in a previous push for an upped version. See here.
However, a more robust and transparent test/system would be better.

[CasperWA]

I would say that our current CI workflow properly tests this - since we do not want to supply testing capabilities with the PyPI package, the test in test_setup.py makes sure all expected non-code files (data files) are packaged into the distribution package.
Feel free to reopen the issue if you feel this is wrong.
Tagging @shyamd as the only other participant in this issue.