What advice do we give maintainers wondering how to best do their jobs now that we're moving away from setup.py?

[njsmith]

With pip 10 imminent, we're finally starting the process of moving away from direct usage of setup.py. In particular, pip 10 will support PEP 518, so pyproject.toml will allow specifying build-system.requires. For projects that use this, the old advice of "just run setup.py ..." doesn't work anymore: now you need to first set up some sort of venv, install the build requires, and then invoke setup.py. And hopefully soon we'll see a pip with PEP 517, where setup.py may not exist at all anymore.

For end-users, this is all fine: pip install <whatever> is the only command they need, and they can forget about setup.py entirely. (And thanks to pypa/pip#536 being fixed, I think pip install . is now a full replacement for setup.py install.)

For maintainers, it's more of a problem. Of course you can do the venv-then-build-require-then-setup.py dance by hand, but that's tiresome, and the whole point of pyproject.toml is that the computer already knows what needs to happen, so asking people to do things by hand is extra annoying. Can we do better?

The core commands that every maintainer needs are: setup.py bdist_wheel, setup.py sdist, setup.py upload. Currently, twine provides an alternative for upload, and there's discussion in #60 of whether this should be folded into pip.

For bdist_wheel and sdist, there isn't currently any direct replacement. pip wheel is kinda a replacement for bdist_wheel, but is awkward because it builds wheels for not just your project, but also all your transitive dependencies, which depending on your dependencies can be time consuming, or fail, and you certainly can't do pip wheel . && twine upload dist/*. And we don't have any setup.py sdist replacement at all.

For these core operations of release (sdist), build (bdist_wheel), and upload, I think we there should be some clear, simple instructions that we can give maintainers and that will work in a uniform way on any Python project in the same way setup.py that invocations currently do.

[dstufft]

I’m torn between wanting to roll twine into pip and wanting to make a twine sdist and twine wheel command that more closely matches the setup.py ones. The second thing might be a good idea though, I would just not have twine worry about setting up the environment at all, it just handles invoking the API.

…

Sent from my iPhone

On Mar 19, 2018, at 7:23 PM, Nathaniel J. Smith ***@***.***> wrote: With pip 10 imminent, we're finally starting the process of moving away from direct usage of setup.py. In particular, pip 10 will support PEP 518, so pyproject.toml will allow specifying build-system.requires. For projects that use this, the old advice of "just run setup.py ..." doesn't work anymore: now you need to first set up some sort of venv, install the build requires, and then invoke setup.py. And hopefully soon we'll see a pip with PEP 517, where setup.py may not exist at all anymore. For end-users, this is all fine: pip install <whatever> is the only command they need, and they can forget about setup.py entirely. (And thanks to pypa/pip#536 being fixed, I think pip install . is now a full replacement for setup.py install.) For maintainers, it's more of a problem. Of course you can do the venv-then-build-require-then-setup.py dance by hand, but that's tiresome, and the whole point of pyproject.toml is that the computer already knows what needs to happen, so asking people to do things by hand is extra annoying. Can we do better? The core commands that every maintainer needs are: setup.py bdist_wheel, setup.py sdist, setup.py upload. Currently, twine provides an alternative for upload, and there's discussion in #60 of whether this should be folded into pip. For bdist_wheel and sdist, there isn't currently any direct replacement. pip wheel is kinda a replacement for bdist_wheel, but is awkward because it builds wheels for not just your project, but also all your transitive dependencies, which depending on your dependencies can be time consuming, or fail, and you certainly can't do pip wheel . && twine upload dist/*. And we don't have any setup.py sdist replacement at all. For these core operations of release (sdist), build (bdist_wheel), and upload, I think we there should be some clear, simple instructions that we can give maintainers and that will work in a uniform way on any Python project in the same way setup.py that invocations currently do. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[njsmith]

Pip is already where all the magic to build wheels lives (including setting up temporary build environments, and in the future invoking PEP 517 backends), so that would be one natural place to put it. I even hear that pip is planning to build an sdist along with each wheel, so I guess we'd get the sdist part for free?

OTOH, pipenv is where we keep the magic for managing environments during development, and invoking tools that use those environments. If we wanted a persistent environment for these builds, maybe pipenv should manage it? (CC @kennethreitz)

Though there's something to be said for constructing a build environment from scratch when making a release.

[kennethreitz]

+1 for twine being built into setuptools :)

[kennethreitz]

implications of that are large, of course, as it has dependencies.

[kennethreitz]

My personal recommendation for the community is seen here: https://github.com/kennethreitz/setup.py/blob/master/setup.py#L49

it works well. Requires twine to be installed to function properly, though.

[theacodes]

I'm a bit on the fence - I'm torn between the Unix philosophy of small tools that do one thing well (I love that twine is basically a glorified uploader) and the fact that unified tools (like npm) are easier for newcomers.

There could be an argument made for either route.

[pfmoore]

On a practical side, smaller tools are easier to maintain, and less prone to problems of maintainers only knowing limited parts of the codebase.

[theacodes]

It's also entirely possible for us to make higher-level tools that string together the lower-level components, see pipenv.

[pfmoore]

@jonparrott Agreed

[ncoghlan]

On the pipenv front, the model I've found to work nicely for library development is to use it this way:

1. You put your library code in a src directory
2. You do pipenv install --editable src at the root of the repo
3. You put your other "things that are useful for working on this project" dependencies in Pipfile (e.g. tox, your preferred linter, etc)

So while it would be scope creep on the pipenv side of things, being able to say pipenv build src or pipenv build . and have it build sdist and wheel archives, ready for a subsequent pipenv upload, seems like a potentially plausible addition to me.

It would still need help from twine and/or pip underneath, though, to actually handle the build step.

(On the "all that and the kitchen sink" front, note the discussion of hatch at pypa/packaging.python.org#375 - I added a reference to it at the end of the dependency management tutorial for the benefit of folks that want a development workflow management tool that's even more opinionated than pipenv)

[astrojuanlu]

Mentioning setup.py test here because I don't think anyone did yet and it's quite common.

[merwok]

That’s true! People including Barry Warsaw tried to make python setup.py test the universal interface to run tests, but I think that it’s mostly tox who serves that role now.

[gaborbernat]

To be universally accomodating (aka for non setuptools builders) all interaction via setup.py should be considered deprecated and not a neat practice. There's no alternative answer and while true tox can be an alternative I would stop from making it the standard (pyinvoke or nox or pipenv or pick any other are great alternatives e.g.). Solving would probably require a further pep on top of 517, which could introduce the idea of test environments and test commands. We don't have anything at the moment though and everyone is free to use whatever they are more comfortable with granted they document how to do it, and is reasonably easy to use.