Writing a setup.py is Hard

[dstufft]

Today most users tend to just cargo cult a setup.py around, never understanding most of the directives that actually go into one.

Solutions:

• Document Creating a New Package From Scratch
• "Package Init", Create a New Package Template?

Some of this already is starting to exist with the packaging guide's page on creating a package (See here), but that is incomplete.

[aaugustin]

Unless the distutils docs drastically improved since I last read them, the missing piece is a focused and correct distutils (or maybe setuptools?) tutorial.

How do I ship properly a dead-simple, Python-only, no-dependencies piece of code? This should fit in one or two screenfuls of docs, and is distinct from the reference docs.

[qwcode]

btw, see https://bitbucket.org/pypa/python-packaging-user-guide/issue/23/stop-recreating-the-pip-setuptools-docs-in

I want to basically dump the "Packaging Guides" attempt at yet another setuptools tutorial.
setuptools needs to own that. It already has one. Let's improve that.

As for distutils, any of it's docs need to be boldly red flagged with "Don't Start Here". Distutils docs need to exist primarily for the api at this point.
https://bitbucket.org/pypa/python-packaging-user-guide/issue/29/modernize-distutils-docs

[ncoghlan]

"Don't start here" is a large part of the PEP 453 plan for the core docs: http://www.python.org/dev/peps/pep-0453/#documentation

Essentially, the distutils docs become a "for packaging tool developers only" reference, while the real cross-version packaging guide becomes independent of CPython.

We may also want to point people in the direction of Audrey Roy's cookiecutter project - it's a nice way to jumpstart a lot of boilerplate packaging and collaboration information.

[ncoghlan]

CPython issue for the PEP 453 "Installing Python Modules" docs update: http://bugs.python.org/issue19407

[ncoghlan]

The distributing Python modules docs aren't in scope for PEP 453, but with the precedent set for the installing side, it's easy to make the case for delegating the build side to external version independent docs as well.

[dstufft]

I've gotten more response about how writing a setup.py is hard with some specific criticism, which included the cargo cult nature of setup.py

[dstufft]

Someone else called out how brittle things are, even a well written setup.py can easily break, sometimes silently, if you rearrange things at all.

[aaugustin]

It's still unclear to me how we went from "never use setuptools, it's too weird, it monkey-patches things to spread itself as soon as it's installed on your system, etc." to "use setuptools". Technically, that must have happened around the setuptools + distribute merge, but isn't setuptools still essentially the same as when its use was discouraged?

From the confused-packager-department... I'm still carefully using distutils.core and not setuptools to avoid inflicting it to users of my packages...

[qwcode]

went from "never use setuptools, it's too weird"

there was propaganda about using distribute over setuptools, because distribute was more open and supported py3, but both still had the monkey patching and were/are "weird.". Just using distutils is a non-starter for many projects, because it doesn't offer "install_requires".

[qwcode]

The big shift for people I think is going from "just wait, the distutils2 revolution is coming in the stdlib", to "distutils2 is not coming, and there is no plan atm to release a replacement into the stdlib, it's ok/safe to use setuptools for packaging/building right now; we're working on it"

[aaugustin]

It's the first time I hear about "distutils2".

Even though I play the devil's advocate in this thread, I'm not totally clueless about packaging.

[qwcode]

It's the first time I hear about "distutils2".

the distutils2 effort (or "packaging", the name it was going to be called in the stdlib) went on for years from roughly 2009 to 2012.

here's a link to a graphic about it in the "Hitchhiker's Guide to Packaging" (which the new "Python Packaging User guide" replaces.) http://guide.python-distribute.org/introduction.html#current-state-of-packaging

[ncoghlan]

The main change was that we (wearing my CPython core developer hat) figured
out most of the really problematic defaults were in easy_install rather
than being inherent to using setuptools on the build side. So
setuptools+pip is substantially better behaved than easy_install (the
latter is hard to change due to backwards compatibility concerns).

setuptools still has a lot of rough edges, but we need it (or an
equivalent) as a low barrier to entry way to produce usable metadata for
pure distutils projects.

[hickford]

You're right, writing a setup.py is intimiding for new users.

Nodejs has a wonderful wizard npm init https://www.npmjs.org/doc/cli/npm-init.html

This will ask you a bunch of questions, and then write a package.json for you.

If you already have a package.json file, it'll read that first, and default to the options in there. It is strictly additive, so it does not delete options from your package.json without a really good reason to do so.

Also Npm's package.json is much simpler to work with than setup.py because it's a lifeless json file rather than a Turing-complete Python script.

The Npm documentation is fantastic too:

• https://www.npmjs.org/doc/misc/npm-developers.html
• https://docs.npmjs.com/files/package.json

[ionelmc]

There's https://pypi.python.org/pypi/d2to1 that would make this declarative (similar to package.json). Why hasn't it gained much traction ?

[alex]

FWIW pbr has run with this idea, and is pretty widely used (all OpenStack
projects use it, and it has about a bajillion downloads).

On Thu, Jul 17, 2014 at 6:50 PM, Ionel Cristian Mărieș <
notifications@github.com> wrote:

There's https://pypi.python.org/pypi/d2to1 that would make this
declarative (similar to package.json). Why hasn't it gained much traction
?

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[dstufft]

On the other hand pbr is wrong and uses a requirements.txt as the source of install_requires :(

[ionelmc]

Last time I checked pbr had some openstack specifics (like that weird OSLO_PACKAGE_VERSION env var thing). It's not readily usable out of openstack - it doesn't even have a link to the source repository in https://pypi.python.org/pypi/pbr ...

[daenney]

FWIW pbr has run with this idea, and is pretty widely used (all OpenStack projects use it, and it has about a bajillion downloads).

True.

On the other hand pbr is wrong and uses a requirements.txt as the source of install_requires :(

This.

I don't think pbr has solved anything. They have a few convenience wrappers but most of it comes from just chucking whatever you were doing in setup.py into setup.cfg, Just have a look at their docs on the usage section: http://docs.openstack.org/developer/pbr/#usage. It's nothing more than the contents of setup.py, just in a different file. It doesn't solve any of the fundamental issues that make setup.py so hard.

[ncoghlan]

Note that one reason not a lot has changed on the build side is because there hasn't been anyone seriously working on fundamental changes to it of late - the install side is hard enough, and until we know what metadata the installation and redistribution tools need, we don't really know what to change on the build side.

Something that's key to eventually eliminating setup.py, though, is relatively complete documentation for the current distutils+setuptools command set. Failing to cover the full scope of what it was nominally replacing is one of the key reasons the distutils2 project struggled to gain acceptance and was eventually dropped.

pip had a narrower scope, in just trying to take out the use of easy_install as an installer, rather than setuptools as a build system, and has been far more successful.

PEP 440 (the new versioning standard) will hopefully be ready for acceptance in the pip 1.6 time frame, and then we can start looking more closely at PEP 426 and 459 for pip 1.7. The acceptance of those should then further decouple the install side of things from the build side, so it will become easier for folks to experiment with new build tools.

[hickford]

I'd like an pip init command analogous to npm init https://www.npmjs.org/doc/cli/npm-init.html

This will ask you a bunch of questions, and then write a package.json for you.

Though this might not be doable until setup.py is replaced with something simpler (static).

[phildini]

I, too, would like something like pip init analogous to npm init. Is there anything more to push for, here? I'm a little sad that the last comment was 2015.

[merwok]

Well pip is the installer, not the build tool. But I think that flit as an init command!

[njsmith]

@phildini The challenge there is that any package init tool is necessarily going to be pretty strongly opinionated, and people have a lot of conflicting opinions, plus the best practices are in a state of flux... but pip init would be a single thing that had to work for everyone, and probably can't change substantially once it's released. So you have to answer questions like: Should it generate a skeleton for an application or for a library? Pipfile or requirements.txt? Do you use tox? Do you provide some test harness, and if so, do you integrate it into setup.py? (Some people think this is a great idea, some people hate it.) Can you even assume that people want a setup.py at all, given that we're working on moving away from them entirely? I can imagine that in a few years a lot of these questions will have settled down somewhat, but right now putting something into pip feels premature.

The most popular solution right now seems to be cookiecutter, and the nice thing about it is that it doesn't have to be one-size-fits-all.

So one thing that might make sense as a short-term improvement would be to advertise cookiecutter at https://packaging.python.org/tutorials/distributing-packages/

[ncoghlan]

While I like the idea of cookiecutter, I've found it personally intimidating whenever I've tried to use it, as the popular templates tend to push you strongly in the direction of doing things right (separate documentation that actively encourages contributions, testing across multiple Python versions with tox, etc). That's a really good thing for professional use, but for personal use it pushed me into "Ugh, this is too much work adding things that I don't personally need, so I'm just going to skip publishing anything to PyPI, and keep this project solely for my own use direct from source control".

flit init does have an interactive setup, and we're working towards getting that to a point where we can recommend it unreservedly to new publishers. That falls into the category of tools that don't expose a human editable setup.py at all though, so the key sticking point to date has been flit's inability to publish sdists that are backwards compatible with widely deployed installation tools (which expect to see a setup.py file).

Still, a combined recommendation of flit init (for "just the basics, thanks") and cookicutter (for "I want to do things differently from the way flit does them") could work.

[phildini]

Fascinating! I had no idea flit existed. I'm pretty concerned about recommending more tools to our already-complex toolchain.

Right now, we officially recommend at least pip and setuptools, but we strongly encourage twine, and we're thinking about encouraging flit. Having to be aware of 3+ tools to create and upload a python package seems like a lot? (not even getting into the issue of getting pip, etc).

I agree that cookiecutter can be intimidating, and I'm honestly not a fan of "download this thing from pip and then point it at an arbitrary github repo", but given that flit doesn't make a setup.py, how would people feel about a PR recommending cookiecutter? Based on feedback here and in freenode#pypa, it seems about 50/50.

[dstufft]

fwiw flit is effectively a replacement for setuptools in the toolchain, not an additional thing.

[njsmith]

@ncoghlan I suppose one could make a "just the basics" cookiecutter and publicize it here. Not sure who would be volunteering to do this though.

[pfmoore]

I agree with @ncoghlan, cookiecutter is quite intimidating in its basic form. However, it would be possible to recommend cookiecutter plus a specific project template that we recommend as a basic form to use. The difficulty is (as always) that there is no consensus on what form a recommended template should take.

(Hmm, I note that @njsmith just said much the same thing :-))

[ncoghlan]

Right, the idea with flit is for it to replace the setuptools + twine part of the current recommendations, so that the baseline recommendations become:

• use pip for package installation
• use flit for publishing pure Python packages

setuptools & twine won't be going anywhere (especially while pypa/flit#119 is still open), but their minimal level of complexity is quite high, whereas flit is more opinionated about how certain things should work (e.g. always getting your source file listing from your version control system, rather than having a separate manifest definition).