[CT-36] [Bug] Airflow v2.1, v2.2 constraints incompatible with dbt v1.0.0, v1.0.1 on codependency click.

[twilly]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

NOTE: This is a minor bug regarding a common integration.

Airflow v2.1 and v2.2 transitively depends on click>=7.1,<9. Airflow requires Flask v1.1 and Flask v1.1.3+ further constrains click to >=5.1,<8.0.

dbt-core 1.0.0. and 1.0.1 depends on click>=8,<9 from commit 11436fe.

The combined click constraint of >=8,<8.0 is unsolvable, preventing the installation of Airflow with dbt v1.0.0 and v1.0.1.

Expected Behavior

The behavior is correct; it's just a nightmare to deal with. IMHO, it can be better by supporting click v7 in dbt v1.0.

Steps To Reproduce

Try installing the latest release of dbt-core and Flask together:

$ pip install dbt-core==1.0.1 Flask==1.1.4

Relevant log output

ERROR: Cannot install dbt-core==1.0.1 and flask==1.1.4 because these package versions have conflicting dependencies.

The conflict is caused by:
    dbt-core 1.0.1 depends on click<9 and >=8
    flask 1.1.4 depends on click<8.0 and >=5.1

### Environment

```markdown
- Python: 3.9
- dbt: 1.0.0, 1.0.1

What database are you using dbt with?

No response

Additional Context

Apache Airflow provides constraint files of known-working configuration and has published constraints that pin Flask to 1.1.4 (e.g. v2.1, Py 3.9). More recent constraints with Airflow v2.2 pin Flask down to v1.1.2 (v2.2, Py 3.9).

I am unsure of Airflow's reasoning behind downgrading to Flask v1.1.2 in v2.2 and the mainline branch, though I suspect it's because Flask 1.1.3 added additional constraints: pallets/flask: Older versions (1.x.x) of flask pin some dependencies in a way that could cause issues. Interestingly David Lord commented that Flask doesn't follow semver though David doesn't mind setting a maximum version on click in Flask v1.1.3 given the v1.1 line is no longer actively maintained.

Airflow is no stranger to transitive dependency problems: old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+). There's some good comments in there and I recommend reading the thread.

Commercial integrations suffer as well. Google Cloud Composer currently ships with Airflow v2.0 and v2.1 and also isn't happy installing dbt v1.0.x.

dbt's constraint on click >=8 may see breakage in pinned Airflow installs, either breaking Flask (implicit <8) or breaking dbt (explicit >=8). Given dbt's constraint is a recent change (Oct 20, 2021 in 11436fe), I'm thinking it may be possible to support click v7 and loosen dbt's constraint.

That's as far as I've dug in and would like some thoughts around dbt supporting click v7. I can make a patch if it'll help the conversation.

[sungchun12]

@twilly Have you tried using the KubernetesPodOperator similar to how gitlab implements dbt Core today: here?

This avoids the dependency headaches altogether and is a common pattern of people I see using airflow and dbt Core.

[twilly]

@twilly Have you tried using the KubernetesPodOperator similar to how gitlab implements dbt Core today: here?

This avoids the dependency headaches altogether and is a common pattern of people I see using airflow and dbt Core.

Good suggestion and thanks for considering my practical needs. I am indeed considering Docker containers to wrap up different tools, including dbt. KubernetesPodOperator fits into that picture, though with its own baggage.

I started digging down this path because Airflow and dbt installed from PyPI into a single virtual environment was intriguing. No containers to build and manage. No calling between different virtual environments. Simple operators from airflow-dbt wrapping the CLI. Looked like a simple and quick way to manage pure dbt workflows. Not wanting to waste an opportunity of "neat, it could just work" go by, I chased this option.

dbt 0.21.1 did easily coexist in the same environment with Airflow compared to dbt 1.0.0+. The difference was this impossible click constraint.

I guess I'm looking for an answer to the question "should dbt play well with Airflow in the same environment?" I lean towards yes because it's a good pairing and the experience of PyPI is nice. But "yes" is not free as it implies reconciling codependencies to the lowest common denominator. I'll probably make a patch on dbt just to see what supporting click v7 looks like and its implications. Actually landing such a change needs more thought. :)

[jtcohen6]

@twilly I just installed click==7.1.2, ran dbt init locally, and ran test/integration/040_init_tests successfully. So I think it would be fine to loosen this requirement to click>=7,<9:

dbt-core/core/setup.py

Line 56 in 1357143

'click>=8,<9',

Is that a change you'd be interested in contributing? If so, there's good cause to make it in a v1.0 patch release, so that folks can upgrade to v1.0 for use with Airflow, and don't need to wait until v1.1. The change feels pretty low-stakes since click is only used by dbt init.

[twilly]

@twilly I just installed click==7.1.2, ran dbt init locally, and ran test/integration/040_init_tests successfully. So I think it would be fine to loosen this requirement to click>=7,<9

Thanks for jumping in! Sorry for being non-responsive here, I was digging around the source code of click and tasks/init.py first, then life decided to be distracting. :)

Funnily enough 040_init_tests does not test integration with click. click methods are patched out, meaning the behavior differences between click versions is not being exercised. I could add a test to verify the default keyword is not provided or set to non-null if provided for the click.prompt call but that may be over-constraining and still not testing for subtle behavior differences. That said, I did test locally by installing click 7.0 and running dbt init. Seems to work as expected.

On a side note, it would be interesting if the integration testing environment did nothing but interact on stdin/out/err. Then you could control environments and run the test suite in different situations.

Anywho, the PR is up at #4681. Give it a gander.

[twilly]

Following up on this, Airflow 2.2.4 and dbt-core 1.0.3 can co-exist with each other in the same virtual environment, at least on my machine with Python 3.9. I'll take that as a win. :)

I'm closing this issue as there doesn't seem to be any obvious installation issues with the two together. That said, I wouldn't recommend making a policy of keeping co-existence a release blocker. It's not an easy thing to track and tackle without significant integration testing and using containers is a more viable, production-quality option. I'd say making the two happy together is an occasional effort thing: tackle co-dependency problems as they arise.

Thanks for the discussion, guidance, and landing of my PR.