Add support for converting pyproject.toml-based Python3 packages. #1982

amdei · 2023-01-03T17:30:48Z

TL;DR:
It works! (With some limitations at the moment though, as this is work in progress currently)

git clone ....
cd fpm
python3 -m pip install importlib_metadata
ruby bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-install-lib=/usr/local/lib/python3.9/dist-packages/ --python-package-name-prefix python3 --python-bin=python3  typing_extensions

Limitations:

Does not support python2 in any way (obviously)

Temporary limitations (TODOs):

Do not support automatic dependencies extraction to the moment
Do not support license extraction to the moment
Checked only on Debian 11 amd64 (arm64 to go) so far
Not sure about applicability to 'native' packages
A lot of @todo's in code
Code quality is poor
Code-style is poor

I had no previous experience with ruby, so any suggestions/comments/guidelines are welcome.

@ObjatieGroba - could you please check Python part of code, and give your suggestions on how to improve it?

Fixes #1780
Fixes #1873
Fixes #1860
Fixes magma/magma#13847

amdei · 2023-01-06T11:16:41Z

After some reflections I've had a second thought.

Standard importlib.metadata is better that backported importlib_metadata
importlib.metadata is still not good enough to obtain all required metadata of not-yet-installed python package
I don't want to build wheel out of downloaded python package
But seems there is no solution without building wheel out of python package prior to converting it to some other format
It is possible and it is easy to get ALL required metadata via wheel API from pkginfo package

Going to revise my solution accordingly.
Stay tuned.

amdei · 2023-02-12T06:48:16Z

It seems to be working a little bit better now.
At least in terms of metadata retrieval.

Things to do:

1. Apply proper arch attribute to generated deb-package. At the moment all toml-based packages has Architecture: all;
2. Instruct wheel to not download dependencies for build package wheel;
3. For some packages fpm now silently dies with zero exit code. Need to get it work again;
4. Deal with packages dependencies. Not easy at the first glance.
5. Speed-up execution on processing toml packages. Now it 3 times slower that setup.py-based;
6. Perform code de-duplication. As now it contains significant amount of redundant fragments.
7. Cleanup code.
8. File some issues to pip, wheel, importlib, etc.
9. Obey --[no-]python-obey-requirements-txt fpm's option

Any suggestions/comments/guidelines are still welcome.
As well as package samples, where things go wrong.

jordansissel · 2023-02-13T02:58:20Z

Thanks for your work on this! I’ll take a look as soon as I can :)

ObjatieGroba · 2023-02-13T14:35:23Z

lib/fpm/package/pyfpm_wheel/get_metadata_wheel.py

+        return deps
+
+    def get_home_url(self, project_urls):
+        res = dict([i.strip() for i in x.split(',')] for x in project_urls)


Theren is no reason to generate dict.
Use next(filter) instead of dict.

return next((x.split(',', maxsplits=1)[1].strip() for x in project_urls if x.lstrip().startswith('Home')), None)

It appears that different packages put home URL into different keys.
For example Home vs. Homepage.

So let's stick to a current approach, if there is no more efficient solutions...

amdei · 2023-02-14T04:38:51Z

Bottom line:

Please update python wheel to latest version before trying anything: python3 -m pip install -U wheel - there was a bug, affected fpm, fixed only very recently.
It is slow. Very slow. Significantly slower that setup.py-based packaging;
Wasn't able to made it with architecture - it looks like there is simply no way to get arch neither from *.whl file nor from sdist via any API. But it contains in *.whl. But I'm not brave enough to parse it in that way (yet).
There is no alternatives found for setup.py's --install-lib, --install-data, --install-scripts and build_scripts --executable. So all fpm's --python-install-* options will not work (and currently silently ignored).
Staging with pip is broken. For example see here: Unexpected --target behavior with/without --upgrade pypa/pip#8799 Or it is intended for something else.
Documentation for pip install is rather short: https://docs.python.org/3/installing/index.html#installing-index Significantly shorter than one would expect.

amdei · 2023-04-08T23:14:18Z

After a while, I hack the Wheel metadata to get information if package is pure or not.
https://peps.python.org/pep-0491/
As it seems to be impossible (at least yet) via pkginfo.Wheel API: https://bugs.launchpad.net/pkginfo/+bug/2015657

amdei · 2023-04-10T00:30:10Z

It reached "works for me" stage.
Let's collect feedback before starting to polish things..

ObjatieGroba · 2023-04-12T20:20:38Z

lib/fpm/package/pyfpm/get_metadata.py

@@ -83,14 +83,17 @@ def run(self):

        if self.load_requirements_txt:
            requirement = open(self.requirements_txt).readlines()
+#            print('REQ-SETUP-PY-REQ-TXT:', requirement, file=sys.stderr)


unnecessary lines

ObjatieGroba · 2023-04-12T20:23:21Z

lib/fpm/package/python.rb

-    if !File.exist?(setup_py)
-      logger.error("Could not find 'setup.py'", :path => setup_py)
-      raise "Unable to find python package; tried #{setup_py}"
+    # @todo Swap it! - TOML priority is for debugging only!


It is time to swap

ObjatieGroba · 2023-04-12T20:23:30Z

lib/fpm/package/python.rb

+      logger.debug("Do job with pyproject.toml")
+      wheel_path = build_py_wheel(setup_py)
+      load_package_info_wheel(setup_py, wheel_path)
+      # install_to_staging_toml(setup_py)


unnecessary comment

ObjatieGroba · 2023-04-12T20:24:45Z

lib/fpm/package/python.rb

+        "#{Shellwords.escape(toml_metadata_code)}"
+
+      # @todo FIXME!
+      #      if attributes[:python_obey_requirements_txt?]


Doesn't this break backward compatibility?

ObjatieGroba · 2023-04-12T20:25:30Z

lib/fpm/package/python.rb

+    ::Dir.chdir(project_dir) do
+      flags = [ "--root", staging_path ]
+
+      # if !attributes[:python_install_lib].nil?


Lot's of commented code. Is it necessary?

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

…and name'.

JordanStopford · 2024-01-25T15:10:23Z

Amy update on this? A lot of packages no longer ship setup.py files so if we need to use fpm then we're stuck on old versions

cwegener · 2024-04-09T01:02:22Z

I don't understand what this does? It still requires a legacy setup.py file to be generated by the python project that is supposed to be converted. However, pretty much all PEP-517 build backends disable the support for generating setup.py legacy files by default, so they will not be able to be used by fpm. (some backends don't even offer an option at all to generate legacy setup.py files)

And if you have a source python project that has the option to generate a legeacy setup.py enabled, then such a project can already be used as a source in fpm without this change.

I am a bit confused what this change is supposed to do.

What is the intention of this change?

amdei · 2024-04-09T16:21:59Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented.
But if there is pyproject.toml file - it will be used instead.

cwegener · 2024-04-09T21:27:51Z

It still requires a legacy setup.py file

No, it doesn't.

There is an option to use setup.py file, if it presented. But if there is pyproject.toml file - it will be used instead.

Thanks for the clarification. I'll need to read through the patch again then. 😁

amdei · 2024-04-17T00:43:36Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

cwegener · 2024-04-17T08:11:38Z

Although in this MR I made attempt to solve the issue via pip wheel, not via build module from PyPA, as suggested in #2040.

That's excellent. pip wheel will actually just invoke the Python build module which takes care of performing the right build process (legacy setup.py vs new PEP-517 build backends)

I will need to test out your PR and report the results.

Specifically, I will need to test with this PEP-517 project ... https://github.com/openai/openai-python

gmabey · 2024-05-08T14:08:22Z

Just an enthusiastic vote for more eyeballs on this merge request ... I would love ❤️ to see the project catch up to this shift in the python community.

cwegener · 2024-05-22T03:20:35Z

I will need to test out your PR and report the results.

Quick feedback from my initial testing of this patch:

Testing with

openai package from PyPI (which uses hatchling as the PEP 517 build backend)
Testing on Ubuntu 22.04 with built-in Python3.10

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel (and to a lesser degree pip download ... which is a separate story).

I think the main concern so far is the new dependency on the pkginfo Python library. E.g. on Ubuntu 22.04, the version of python3-pkginfo is a bit older and does not yet have support for version 2.3 Metadata file formats.

But a lot of wheels will now write 2.3 Metadata file formats.

Therefore, injecting or vendoring an up-to-date version of pkginfo is important.

amdei · 2024-07-14T20:20:22Z

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building?
Is it difficulty with determining build dependencies?
Or something else?

cwegener · 2024-07-14T21:31:28Z

My main issue that I'm looking into at the moment is the boot-strapping of the PEP 517 build environments for pip wheel

@cwegener could you please elaborate, what is your issue with openai package building? Is it difficulty with determining build dependencies? Or something else?

I think it's actually a different issue that is unrelated to the PEP-517 build support.

The issue is that all dependencies are being built from source due to a change introduced here: 77e0cf7

The --no-build :all: option forces all dependencies to be built from source, even though this is unnecessary.

The dependencies will be thrown away at the end. They don't end up in the released package that fpm creates. So there is really no benefit in building them from source.

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

amdei · 2024-07-14T22:17:14Z

Nevertheless, I still have not managed to build an openai package, but I think that might just be an issue on my side.

Just checked - built seamlessly.

Could you please try to build it with more verbose output and share the result?
Something like this:
./bin/fpm --verbose --log debug --debug --debug-workspace -s python -t deb --python-bin=python3 --python-package-name-prefix python3 --python-install-lib=/usr/local/lib/python3.9/dist-packages/ openai

(you may want to adjust --python-install-lib option to reflect your python version)

cwegener · 2024-08-20T06:13:21Z

I had another look. And I think I figured out where the issue with this approach is.
pip download will perform unwanted build steps in order to generate metadata. Those unwanted build steps will fail quite often.

Example package: jiter

python3 -m pip download --no-clean --no-deps --no-binary :all: -d . jiter

This will try to immediately build a wheel from the jiter sdist, which will fail due to the build backend (maturin) not being present on my system.

This is a pip problem as per the following references.

References:
pypa/pip#7995
pypa/pip#1884

amdei · 2024-08-28T23:14:06Z

And I think I figured out where the issue with this approach is.

Could you please elaborate, do you able to offer different approach?
Yes, maturin, for example, requires Rust in order to compile its dependencies.

May be you know the way to download and use pre-build binary packages, for all possible supported OS/architectures?

gmabey · 2024-09-18T22:42:09Z

@amdei This branch worked for me on ubuntu 24.04!
I did have to install some packages that probably already should have been installed: python3-pkginfo, python3-packaging, python3-setuptools and python3-wheel. It looked to me like something was trying to automatically install some of them but that effort (whatever was doing it) produced errors. However, when I just installed the distro-provided versions, everything worked!

On ubuntu 22.04, ubuntu 20.04, rockylinux 8 and rockylinux 9, I had to both upgrade the distro-provided pip as well as upgrade the above modules before the creation of a .deb from python/pip would work.

So, I conclude that the core of your patch works great, but there may be some dependency auto-foo that is lacking?

Thank you for submitting this MR!

CommandMC mentioned this pull request Jan 10, 2023

Update GH actions & specify Python version as dependency derrod/legendary#537

Merged

amdei force-pushed the python2deb-great-again branch from 419d841 to 9e16f11 Compare February 12, 2023 03:54

ObjatieGroba reviewed Feb 13, 2023

View reviewed changes

bugfood mentioned this pull request Mar 13, 2023

PEP518 - setup.py is deprecated #1987

Open

amdei marked this pull request as ready for review April 10, 2023 00:29

ObjatieGroba reviewed Apr 12, 2023

View reviewed changes

amdei added 14 commits September 23, 2023 01:31

Add support for converting pyproject.toml-based Python3 packages.

8d88bbc

Fixes jordansissel#1780 Fixes jordansissel#1873 Fixes jordansissel#1860 Fixes magma/magma#13847

Works again, with metadata from wheel.

c1b7e17

Imporve metadata parsing.

3cf33e8

Increase TOML priority over setup.py - for debugging only!

f94ef4e

Do not build wheels for dependencies.

fa08e5f

Fix python packages install flags in case of toml.

b3a66ce

Attempl to deal with dependencies correctly.

3df693a

Add check for python Wheelness.

c0c0a8a

Drop check for importlib.metadata python package.

a1777e6

Geting things easier.

c66239b

Fix staging install with --ignore-installed for pip install.

79411b1

Obtain package architecture via hackily hack.

451bb62

Fix native packages detection.

7e4fcac

Fix 'ERROR: The --python option must be placed before the pip subcomm…

43fefae

…and name'.

amdei force-pushed the python2deb-great-again branch from a9a1434 to 43fefae Compare September 22, 2023 22:49

amdei mentioned this pull request Nov 19, 2023

could not find setup.py in installable package #2029

Open

Stealthii mentioned this pull request Apr 24, 2024

PEP517 Support #2056

Open

Merge branch 'jordansissel:main' into python2deb-great-again

34e2b16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for converting pyproject.toml-based Python3 packages. #1982

Add support for converting pyproject.toml-based Python3 packages. #1982

amdei commented Jan 3, 2023 •

edited

Loading

amdei commented Jan 6, 2023 •

edited

Loading

amdei commented Feb 12, 2023 •

edited

Loading

jordansissel commented Feb 13, 2023

ObjatieGroba Feb 13, 2023 •

edited

Loading

amdei Apr 8, 2023

amdei commented Feb 14, 2023 •

edited

Loading

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

ObjatieGroba Apr 12, 2023

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 •

edited

Loading

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

amdei commented Jul 14, 2024

cwegener commented Jul 14, 2024

amdei commented Jul 14, 2024 •

edited

Loading

cwegener commented Aug 20, 2024

amdei commented Aug 28, 2024

gmabey commented Sep 18, 2024

Add support for converting pyproject.toml-based Python3 packages. #1982

Are you sure you want to change the base?

Add support for converting pyproject.toml-based Python3 packages. #1982

Conversation

amdei commented Jan 3, 2023 • edited Loading

amdei commented Jan 6, 2023 • edited Loading

amdei commented Feb 12, 2023 • edited Loading

jordansissel commented Feb 13, 2023

ObjatieGroba Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

amdei Apr 8, 2023

Choose a reason for hiding this comment

amdei commented Feb 14, 2023 • edited Loading

amdei commented Apr 8, 2023

amdei commented Apr 10, 2023

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

ObjatieGroba Apr 12, 2023

Choose a reason for hiding this comment

JordanStopford commented Jan 25, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 9, 2024

cwegener commented Apr 9, 2024

amdei commented Apr 17, 2024 • edited Loading

cwegener commented Apr 17, 2024

gmabey commented May 8, 2024

cwegener commented May 22, 2024

amdei commented Jul 14, 2024

cwegener commented Jul 14, 2024

amdei commented Jul 14, 2024 • edited Loading

cwegener commented Aug 20, 2024

amdei commented Aug 28, 2024

gmabey commented Sep 18, 2024

amdei commented Jan 3, 2023 •

edited

Loading

amdei commented Jan 6, 2023 •

edited

Loading

amdei commented Feb 12, 2023 •

edited

Loading

ObjatieGroba Feb 13, 2023 •

edited

Loading

amdei commented Feb 14, 2023 •

edited

Loading

amdei commented Apr 17, 2024 •

edited

Loading

amdei commented Jul 14, 2024 •

edited

Loading