Autogenerate SpineOpt System Image based on tests

[DillonJ]

In the last SpineOpt developer call - we discussed performance a little.

One of the ideas was helping the user by making it easier to generate a system image that touches as much SpineOpt code as possible. @manuelma's idea was to generate a system image based on the tests.

I think it would be great if we went just one step further and provided users with pre-compiled system images for each supported platform based on the tests.

This would provide users with compiled code performance and the best possible experience of SpineOpt where the first time model-build performance is clearly an issue. This would go a long way to addressing that.

I think this is the single best thing we could do right now for performance.

We could also provide a SpineOpt plugin that uses this image by default.

Might this also make deployment a little easier?

Thoughts @manuelma @jkiviluo @datejada

[manuelma]

This is a great idea and a good reason to keep coverage high. I think we touch the most common constraints at the moment so should be good.

We can start with a couple of platforms, Windows 64 and Linux, and see how it goes.

The plugin idea is also good, we can provide an Executable tool specification that runs SpineOpt from the binary, and then as @DillonJ says they don't even need to install julia.

We need a little bit of muscle though: somebody to generate the images and then some others to test them in their systems. Then we need to figure out the deployment, how do we distribute the binary and that kind of stuff.

[abelsiqueira]

I don't know much about system images to know how feasible or easy this would be. A quick search showed that someone tried something similar: https://discourse.julialang.org/t/system-image-as-an-artifact-download-pre-build-system-images/44106
But it was some time ago, and their application was different, and smaller in scope (looks like a light CLI around Pkg.jl).

[manuelma]

Pretty much everything can be automated via github actions. In principle, we'd just need an action that runs the tests with --trace-compile=file.jl and then calls create_sysimage(..., precompile_statements_file=file.jl).

The devil is in the details though and I feel there will be a lot of those. Provided everything works, we need a place to put the image so users can download it, etc. - all the deployment needs to be figured out.

[abelsiqueira]

If building the image succeeds, then I think uploading it as an artefact to the release is not too hard - in comparison, at least.

[suvayu]

Artifacts on GH have an expiry. We have to have a policy on how long is the correct duration (can't be a yr, I think the max is several months). Typically long lived artifacts are uploaded to something like bintray, but don't know if they have a free tier.

Also not sure about size limits. If it's a few MBs, no big deal. But several hundred MBs would be a problem.

[DillonJ]

Feels like a place to start would be to try and do this manually and test the system image on a few models and compare to the first time run performance and second time run performance to see what benefit we would get on average

[jkiviluo]

Sounds all very good. I just wonder whether there is a more systematic way to include all constraints in the system image than relying on the tests. Of course it would be good to have full test coverage, so maybe it's the way go in any case.

[DillonJ]

Yeah - it feels like the lowest effort way and a good starting point - but an interesting and valuable task would be to create the smallest model or set of models that use the most constraints/functionalities

But I think the test-based approach would be getting us the 80/20, almost quite literally since test coverage is around 80% (actually it's zero now - something must be failing - but was 86%)

[jkiviluo]

I was more wondering if there is a way to force all code in the package to be part of the image without having to define data to capture all corners.

[DillonJ]

That's the thing about SpineInterface though - the data defines the code

[manuelma]

It's not due to SpineInterface. It's just that Julia doesn't compile the code unless it's executed. So you need a way to execute your code in order for julia to do the compilation.

[jkiviluo]

Right. And apparently there is no switch to force compiling even when code is not executed.