Unpin requirements.txt

[jorisroovers]

This issue can be used as a continuation of #133 which I inadvertently (and irreversibly) closed when deleting the master branch in favor of the new main default branch.

Please read ALL of #133 before commenting here.

[MRigal]

Hi @jorisroovers

First, thanks for this powerful package!

I would actually rename / invert the title and logic desired here.

Unfortunately, constraining the packages pinned in the setup.py is not only bad practice as defined by pip, but creates also a lot of pain to many people when using gitlint with other dev tools. Not only with black, but it is actually unusable (because of incompatible versions) with most packaging tools like poetry or pip-tools if you have some other libraries defining click, arrow or dateutil..

Thus my suggestion would be this one:

1. Keep in requirements.txt strictly pinned dependencies, for running the tests and having a "working reference". You could actually use pip-tools to pin and update the versions more easily.
2. Have a pretty lax setup.py file (which is the one causing the conflicts), something like Click>=7 and arrow>=0.15
3. Have a mention in the docs (FAQ ?) that gitlint might produce different behaviour depending on the version of the libraries used, referencing the requirements.txt as a possible example

What do you think about it?

[andersk]

I’ve proposed a possible compromise solution in #220: provide the pinned requirements with pip install gitlint[pinned], and document that as the supported way to install gitlint, but also allow installation with unpinned requirements using pip install gitlint.

[sigmavirus24]

I mean pinned is such an understated extra name for this. Perhaps gitlint[trusted-deps] or something that more clearly communicates "This is what we've tested". Regardless, your solution opens the project up to countless bugs of "gitlint doesn't work with {Dependency} version {X} that was released 5 minutes ago and has backwards incompatible changes" which it neither has the appetite nor the time for

[andersk]

I understand the underlying frustration. IMO, it should be directed at the Python packaging system. Python does not allow installing multiple versions of a package, the way (say) Node.js does—in Node.js you could have different versions at node_modules/click and node_modules/gitlint/node_modules/click and nobody would complain.

But in Python, single versions are the reality we have to deal with, so most Python package maintainers will need to deal with a few bug reports about version incompatibilities.

• If you don’t pin your requirements, you might get bug reports when there are backwards-incompatible changes in your requirements.
• If you do pin your requirements, you might get bug reports when there are conflicts with other packages that require newer versions.

Receiving these bug reports doesn’t mean your software is bad—it’s just part of the job description of a Python package maintainer.

Although neither type of bug report may be theoretically avoidable, in practice, backwards-incompatible changes to Gitlint’s requirements have never affected Gitlint: if you remove the pins, the very newest versions of Gitlint’s requirements seem to work fine with the very oldest version of Gitlint that supports Python 3 at all. (I understand that some of the tests are more sensitive to the precise versions, but I’m not proposing to allow unpinning the versions used for running tests.)

But version conflicts plainly do affect Gitlint: there were many complaints when Gitlint could not be co-installed with Black, and right now Zulip is unable to use the current version of Gitlint because it cannot be co-installed with the current version of Semgrep.

By its nature as a linter tool to be used within other projects, many of which are also written in Python, Gitlint is most useful when it places as few restrictions as possible on the Python libraries it can be co-installed with. On the other hand, I understand the advantage of providing a recommended set of known-good versions. I’m trying to find a way to deliver this advantage while acknowledging the reality that users sometimes need to override these recommendations. No solution is perfect, but I’m hopeful that using an extra will work well enough.

As far as extra names, I’m happy to go with whatever name you decide on.

[jorisroovers]

I want to re-emphasize that my position on this has little to do with what is the right/wrong technical decision. I think you make a strong argument @andersk, but in the end it all comes down to the same thing: my availability to timely merge changes and push releases out. That is not guaranteed at all (see #134 for additional context). As a result, I've made a decision that I believe will cause me the least pressure.

I do like the idea of using extras as a way to facilitate us both - that hadn't occurred to me.

However, for me to be comfortable with it, I'd want the logic to be reversed: use pinned dependencies when using the 'default' pip install gitlint and providing an alternative command (e.g. pip install gitlint[loose-deps]) to install gitlint using less strict dependency requirements.

Having a quick look at setup.py docs, I'm not sure whether that's actually possible. Maybe not using install_requires at all but just 2 versions of extra_requires, one that is defaulted but can be overwritten/deselected? Again, not sure if possible, would need to try it out - appreciate any help!

[andersk]

Unfortunately, the Python packaging system doesn’t support extras that loosen dependencies rather than tightening them, nor does it support default extras (pypa/setuptools#1139).

A possible workaround would be to rename the main package to (say) gitlint-core, and have gitlint be an empty package that just requires gitlint-core[trusted-deps]. But I’m not sure if maintaining two packages rather than one would be an improvement from your perspective, even if one is empty?

Alternatively, what do you think about simply adding a question to the GitHub issue template asking whether the user installed with pip install gitlint[trusted-deps] as documented, and/or asking for the output of pip freeze?

[timabbott]

@jorisroovers is there any chance you can resolve this, through one of removing the package pinning, merging a version of #220, or if you don't have time, through giving @andersk the necessary permissions execute a solution you're happy with?

If this can't be resolved in the next few weeks, Zulip will likely need to either fork Gitlint or stop using it, neither of which feels like a happy resolution to this packaging issue. (We're currently working around this problem by using the Ubuntu package rather than using gitlint via pypi, but we need our development environment to work fully on Debian 10, which doesn't have said package).

[jorisroovers]

Ok, no guts, no glory - so let's try something :-)

I've actually been pondering about @andersk suggestion of creating 2 packages, gitlint and gitlint-core[trusted-deps]. I think it's an elegant solution with only a small overhead at release time and the least potential of creating fire-drill scenarios for me.

Any suggestions on how to best do this? Creating a separate repo or creating 2 separate packages from this source code repo? I've never done the latter. I want to avoid renaming this repository.

On timeline, depends on how much work I need to do myself. Appreciate PRs :-)

@sigmavirus24, please chime in.

[sigmavirus24]

I've actually been pondering about @andersk suggestion of creating 2 packages, gitlint and gitlint-core[trusted-deps]. I think it's an elegant solution with only a small overhead at release time and the least potential of creating fire-drill scenarios for me.

To summarize this suggestion:

• gitlint would break often (as opposed to now where it's rock solid and functional if used with best practice tools like pre-commit and pipx which isolate it) because the dependencies of the gitlint package would be unpinned
• gitlint-core[trusted-deps] would require folks wanting to use a trusted configuration to update their requirements etc.

I'd argue that the solution should be the inverse:

• gitlint has a dependency on gitlint-core[trusted-deps]
• Users who want to embrace chaos and instability should change their requirements to gitlint-core and then explicitly opt into that behaviour.

I've done something similar to this in the past. uritemplate.py and uritemplate were created roughly at the same time and had the same import namespace. Eventually the RFC authors who implemented uritemplate were no longer maintaining it so we merged them together and I retired uritemplate.py effectively. I uploaded one more version that declared a dependency of uritemplate >= x.0.0 and I haven't updated uritemplate.py since. We could do something similar with gitlint. We could release:

• gitlint 1.0.0 with a dependency of gitlint-core[trusted-deps] >= 0.18.0 (or whatever the real version numbers are) and not have to rev gitlint's version number.
• Only release gitlint-core and bump that version number.

If we want to be more conservative

• gitlint 1.0.0 could have a dependency of gitlint-core[trusted-deps] < 1.0 and then when we bump gitlint-core we bump gitlint (specifically for python version classifier, python_requires etc. changes for minimum supported versions of the interpreter)

As for practical implementation:

We could use 2 repositories. There's not much reason to do that though, so we could instead have 2 package directories.

gitlint/
- setup.cfg
- setup.py
gitlint-core/
- (current gitlint py files)
setup.cfg
setup.py

Otherwise, 2 repos could work well. We just need to figure out how to redirect people for PRs and other contributions

[andersk]

To summarize this suggestion:

• gitlint would break often (as opposed to now where it's rock solid and functional if used with best practice tools like pre-commit and pipx which isolate it) because the dependencies of the gitlint package would be unpinned

@sigmavirus24 No, this suggestion was exactly what you describe as “the inverse”, where gitlint requires gitlint-core[trusted-deps].

Anyway, it sounds like we’re on roughly the same page, so I’ve put together a PR: #246.

[jorisroovers]

As we're getting close to getting #246 merged, I wanted to leave a note of appreciation for @andersk for getting us out of this impasse. You both came up with multiple suggestions (after taking the time to understand my perspective), and then also followed through and did the implementation. Thank you - gitlint is better because of it!

And @sigmavirus24 of course, for perspective and expertise (always :-) )

Thank you!

[jorisroovers]

Just released 0.17.0 with this change.

For others referencing this issue in the future: you can now install gitlint with unpinned dependencies using:

pip install gitlint-core

@andersk I tested it myself, but appreciate if you can just confirm this is working from your end as well. Thanks!

[andersk]

Looking good from here. Thanks so much!