Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modernize packaging #278

Open
sneakers-the-rat opened this issue Aug 10, 2024 · 0 comments
Open

modernize packaging #278

sneakers-the-rat opened this issue Aug 10, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@sneakers-the-rat
Copy link

Is your feature request related to a problem? Please describe.

saw someone recommend this tool, and some labmates were thinking of using it. noticed that install is only possible with docker or conda, and went to see why. the package and its deps aren't terribly complex, but it seems like there is just a lot of tech debt spread out across a number of packages that make it so eg. this package isn't on pypi and isn't as simple to use as pip install moseq2-app && moseq2

other problems:

  • looks like a lot of places in the docs have to instruct users to ignore warnings, and resolving those warnings requires updating package versions and code, but that's not really possible to do since things aren't packaged tidily, and there isn't a CD action to make it easy to update for everyone if it did happen. this tech debt only gets worse over time :(
  • the versions that are pinned in this project and its dependencies are very old, eg. numpy 1.18.3 was released more than 4 years ago. python 3.7 reached EOL status more than a year ago, and is no longer receiving security patches. outdated software, particularly high use software like numpy and python, is unsafe to use and should be updated frequently.
  • setup.py based installations are slow and dangerous - to even check the package metadata, the setup.py file has to be run, so the python world is moving away from them. eg. this package downloads and extracts a tarball to a deps directory under the working directory when being installed - so if a secondary package were to try and depend on it, any time it was installed that directory would be created no matter where the command was run! thankfully there is no reason to use setup.py anymore in a post PEP 517/518/621 world :)
  • it's effectively impossible to develop downstream tools when the only means of use are docker containers/conda forge packages or installation has to happen from the git repository - i can't add this as a dependency in my project and use components from it, which enriches the scientific software ecosystem as a whole (as well as extends the use and usefulness of your work!) unless i too make a docker image. the exact version pins make this problem worse because then every downstream package has to have those same outdated version pins.
  • it's harder for y'all to develop the tool and make regular releases because the tech debt adds up and each change becomes a harder fight against a sea of outdated dependencies. it might be nice to use some feature from a newer package, but that can't happen if it requires a version of numpy > 1.18.3. I see changes being made in the repo as recently as 3 months ago, but the last released versino was april 2023, same with the docker image.
  • it ends up making labor get duplicated, like i see keypoint-moseq doesn't depend on or use code from this set of packages, so while all of them seem like great tools, they don't build cumulatively and stay maintained, and so a lot of work has to be done twice and labs that use the tools get stuck in a lurch!

fix

it's not that complicated, but since there is a relatively long chain of deps here that need to be fixed, and some look like they come from outside of the lab and may be abandoned, it requires code to get touched in a bunch of places.

Here i've updated all the relevant packages to modern packaging tools so that they can all go on pypi and be reused and maintained :)

all of these correctly install (and compile, if they use c extensions), and i got to the point of starting to update calls from outdated packages but stopped to raise this issue and let you know that they are in a state where they can be worked on if you want the changes. i can PR all of these if you'd like. For the packages that aren't from the lab, i could be wrong but most of them seem like they're MIT licensed, so we can just make secondary updated pypi packages for them since they haven't been updated in 4-5 years anyway.

The tests currently don't pass because of those outdated call issues, but there was a module import in the tests of a module that doesn't seem to exist anymore, so pytest couldn't even start collection, so it seems like the tests maybe don't get run all that often. we can fix that too and ensure that all code always gets tested before committed to default branches :).

we made some nice guides over at pyopensci - https://www.pyopensci.org/python-package-guide/ - and i would be happy to help with the transition since this seems like a relatively common tool in neuro, and software maintenance is a big, hidden part of what makes the work possible. thanks to the group for writing these tools! <3

@sneakers-the-rat sneakers-the-rat added the enhancement New feature or request label Aug 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant