Run the test suite as a GitHub Action

[EmperorArthur]

No description provided.

[pimlie]

This could be just openscad, no need to add the /bin/ prefix as Linux resolves the path by whatever you put in env.PATH

[EmperorArthur]

Does it in cases where you're not running through a shell?

Edit: I also don't think the automatic expansion works when doing the exists() check.

[Ruudjhuu]

I am a fan of the absolute path as it won't depend on the users environment. If something is by default on Linux it does not mean we need to depend on it when it is not necessary. Defaults van change, users can do weird shit. We already have a big dependency on openscad. If they change the install location, it fails everywhere and we can change it.

[pimlie]

This most likely installs the v2021 version, which is quite old. It might be beneficial to either install/use a nightly (appimage) version or even better use openscad's docker container for these tests.

[Ruudjhuu]

I do agree it is very old but it is the latest stable afaik.

Maybe it is possible to use a fixed "nightly" version? I hate the ci/cd breaking due to the nightly being updated and unstable. I also don't like an openscad binary in this repo.

[pimlie]

FWIW, I had also been working on adding a github action. You can see my WIP branch for that here: https://github.com/kennetek/gridfinity-rebuilt-openscad/compare/main...pimlie:test-use-bash-github-yaml?expand=1 It's still a WIP as I was testing the workflow with act but ran into an issue with the work folder.

Not sure if it's worth continuing that branch, but to explain my choices:

• I used bash and not python so we could run tests directly using the openscad docker image once built-in EGL support lands (see remark on https://github.com/openscad/docker-openscad under 'Rendering a PNG).
• The Python tests contain a lot of repetition, but due to way the tests are setup using the parameterized .scad files, we basically only ever need to change the scad parameters and maybe camera angle. Therefore I opted to use yaml files to configure the tests as that has a much lower threshold for adding new tests
• I was planning to adding support for image comparison similar to openscad does

But I will hold off on working on my changes for now as to now 'compete' with the work from this pr.

[EmperorArthur]

@pimlie I suspect it's partly also a methodological approach.

• I chose Python's unittest because it's a simple test framework that's easy to install and run. I'm not opposed to other test frameworks at all, but feel that trying to homebrew a testing framework just to avoid a dependency causes far too many problems.
• Do your tests run under Windows? It is very common for git bash to be installed, but it's not required for someone to use git.
• You might notice the Python tests can also use scad parameter files for some tests. The difference is instead of one file per test, the parameter being tested is modified.
• This is explicitly doing unit tests at the library level. Hence all the pictures of hole negatives.
• Image comparison would be nice, but I wonder if we could use csg export. No major feelings either way.

Future Work

• Realistically, the hole tests should be changed to do a matrix setup instead of brute force copy/paste,
• I use Python's unittest framework not because it's the best test framework but because it's part of Python, and that is the easiest cross platform tool I am familiar with.

[pimlie]

@EmperorArthur

• RE: Do your tests run under Windows? -> They should under WSL. Native Windows support itself for tests should not be a strict requirement imho, too many possible caveats/limitations. The main env (and thus first priority) that will run the tests should be the github workflow which normally runs directly under Linux, if we can manage to run the tests under Windows too then that could be a nice-to-have but requiring Windows test support could limit the project.
• RE: You might notice the Python tests can also use scad parameter files for some tests -> I did indeed and thought that having a default parameter file and then overwrite individual parameters on the cli was an excellent choice by you for the tests! Hence why I tried to copy that behaviour, I just used yaml to make that 'overwrite' behaviour super clear and simple to do :)

Btw, I looked at quite a bit of other SCAD projects and I didn't find any other projects yet that have fully automated OpenSCAD tests. Have you found any? Cause it would also be nice if we wouldn't have to (re-)invent the wheel for that...

Anyway, don't let my remarks/suggestions keep you from implementing the tests the way you feel is best :)