Please don't use setup.py (and some other dependency-related thoughts)

[bswck]

Feature category

Dependency management.

Problem description

Using setup.py should generally be avoided if there are some declarative options.

For this project, I see numerous alternatives. https://github.com/CERT-Polska/mwdb-core/blob/16bbb2cf15f7401391602376d85aa0f6c5f3eabc/setup.py can be easily rewritten entirely into a pyproject.toml file that follows the packaging guide strongly recommended by PyPA.

Moreover, setup.py is known for posing some serious security concerns, such as the ones covered in this pretty interesting "TOP 8 Malicious Attacks Recently Found on PyPI" article. Many of those attacks simply base off on that simple mechanism of setup.py running arbitrary code on the user's end.
This might be concerning especially nowadays, when some people use ChatGPT and similar chatbots to write code for them, but those make up library names that can later on become malware (Mateusz Chrobok had a good take on that).

Some projects do have a reason to run some code that is needed to completely build the project, like in pandas, or to navigate a user better to improve the DX, like in Django, but it's not like that in this case.

The current requirements.txt file doesn't allow security updates. Considering the fact that most of these packages seem to follow the SemVer convention, it would be nice to allow the latest PATCH versions (excluding those dependencies that don't follow SemVer—are there any?).

The solution I'd like

While jumping between branches, I saw this branch https://github.com/CERT-Polska/mwdb-core/tree/use-poetry with commits from @psrok1. Poetry's pyproject.toml does not yet conform to PEP 621, but this seems to be currently worked on. What is the status of that initiative of migrating to Poetry?

Also considerable

Simply migrating to a declarative, pyproject.toml-based dependency specification, and using setuptools_scm, hatch or really any other suitable backend.

[bswck]

To get started, check out https://packaging.python.org/en/latest/guides/modernize-setup-py-project.

[psrok1]

Hi! Yeah I looked a bit into more modern setup e.g. our https://github.com/CERT-Polska/lint-python-action conforms the latest pyproject.toml recommendations. Unfortunately in mwdb-core this may require a bit more work considering plugin compatibility.

The current requirements.txt file doesn't allow security updates. Considering the fact that most of these packages seem to follow the SemVer convention, it would be nice to allow the latest PATCH versions (excluding those dependencies that don't follow SemVer—are there any?).

Unfortunately, not everyone is strictly following the SemVer convention e.g. Pallets packages has a lot of stories with removing deprecated elements in MINOR releases (while breaking changes should be done in MAJOR) and they sometimes doesn't pin their own dependencies properly: #543

In terms of security updates, I try to rely on dependabot here but I understand it might be difficult to upgrade dependencies for an older release of MWDB Core with such strict pinning.

So I'll definitely look into it, but I would like to experiment with different build systems on more simple projects first like Karton or malduck.

[bswck]

Sure. Let me know if I can help.