Build isolation and pinned versions

[nijel]

Description

This is more a question about best practices, originally discovered at xmlsec/python-xmlsec#200.

• python-xmlsec has specified PEP 518 built-time dependency on lxml>=3.8
• the current virtualenv has installed lxml==4.6.5
• pip install xmlsec==1.3.12 will result in xmlsec being compiled with lxml==4.7.1 (because it is currently the latest version), but it will be running with lxml==4.6.5

This can be a source of problems as this is C compiled code and when some interface changes, it can lead to crash.

This was discovered when broken 4.7.0 lxml was released (it is now yanked) and pinning lxml on an older version didn't fix the build.

Expected behavior

Same lxml being used for both building and running.

I can see the following approaches to that:

• switch to using legacy setup.py based build without isolation; switching back to legacy solution doesn't sound good
• pin exact lxml version in both runtime and build-time deps; this will not be really welcome by the users
• compare lxml versions at runtime and fail in case they differ; this will make installation problematic, forcing users to disable build isolation and install build-deps manually
• let pip install the same version of the package in both environments when possible; this might be tricky as the runtime environment

pip version

21.3.1

Python version

3.9

OS

Linux

How to Reproduce

1. pip install lxml==4.7.0
2. pip install xmlsec==1.3.12
3. The build should crash as lxml 4.7.0 is missing some files needed for build. Instead 4.7.1 is used for build.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

You can use the PIP_CONSTRAINTS variable to provide a constraints file.

However, this is broadly a part of #4582, so I'll consolidate this issue into that.

[nijel]

It's not realistic to expect users to do something beyond pip install. This is especially true for leaf packages and cases where the additional constraint comes from some other module.

For example, pip install 'Weblate[SAML]==4.9.1' will install lxml==4.6.5 (because Weblate constraints the version), but the installed xmlsec==1.3.12 is built against lxml==4.7.1.

[pfmoore]

@nijel On the contrary, it's not reasonable to expect pip to be able to detect every possible form of undeclared dependency without some help from the user.

Build dependencies and runtime dependencies are almost always unrelated. A case like this, where a library can only run with the precise version the library was built with is extremely rare (even numpy, the usual example where people hit issues like this, maintains some level of backward compatibility at the binary level). So pip can't be expected to know that there's such a tight dependency without some information from the user, or the library author.

It's possible that the library could handle this via some sort of build-time generation of dependency metadata. Giving the user a way to specify is therefore mainly a workaround for incorrectly specified library metadata.

[nijel]

Luckily, there seems to be binary compatibility for now, but I don't think there is any guarantee on that. That's why I'd like to have a way to address this. But it has been already widely discussed in #4582 and https://discuss.python.org/t/support-for-build-and-run-time-dependencies/1513/80 (unfortunately I didn't find it before).

xmlsec could probably dynamically generate dependencies when building the wheel, but I'm afraid this will not really work - in case it would be incompatible with what is currently installed, pip will keep trying older versions which do not have this constraint, so instead of having current xmlsec with mismatch in lxml, user will end up with having old xmlsec and the mismatch in lxml will stay.

[pradyunsg]

See #10655; after which it can start rejecting to build in certain situations.

[pradyunsg]

Anyway, if you have something to add, please do so over in #4582. That's currently the catch-all for this broader topic