Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-3438] UserConfig in profiles.yml -> flags in dbt_project.yml #9183

Closed
1 task done
Tracked by #9099
jtcohen6 opened this issue Nov 30, 2023 · 16 comments · Fixed by #9289
Closed
1 task done
Tracked by #9099

[CT-3438] UserConfig in profiles.yml -> flags in dbt_project.yml #9183

jtcohen6 opened this issue Nov 30, 2023 · 16 comments · Fixed by #9289
Assignees
Labels
enhancement New feature or request flags user docs [docs.getdbt.com] Needs better documentation

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Nov 30, 2023

Housekeeping

  • I am a maintainer of dbt-core

Short description

We are deprecating config as a thing that is settable in profiles.yml. Instead, we will support flags in dbt_project.yml

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

  • If they have, continue to respect it, and raise a deprecation warning.
  • If they haven't, great. Instead, we will be reading flags from dbt_project.yml. The two are mutually exclusive.

In a future release, we will remove UserConfig from profiles.yml entirely.

Acceptance criteria

In dbt_project.yml, it will look like:

# dbt_project.yml
flags:
  send_anonymous_usage_stats: True

Going forward, some flags will be settable in:

  1. "Project flags" (dbt_project.yml) only. These flags define behaviors tied to the source code in a project, and should remain consistent across all users, environments, and invocations of that project.
  2. CLI option / env var only. These flags are highly specific to the running invocation, and don't make sense to define in version-controlled code. Several of these are file system paths, such as --project-dir or --log-path. Some want to vary across commands, such as --full-refresh. Environment variables are best suited to flags that want different settings in different environments, such as turning DBT_FAIL_FAST on in dev/CI but not in prod.
  3. Both places. In this case, a project wants to set a default behavior, but it may be appropriate to override it for different environments/invocations.

If set in both places, the inheritance hierarchy remains the same:

  1. CLI var (handled by click)
  2. Env var (handled by click)
  3. User config Project flags

Implementation Notes

  • We should rename the internal UserConfig to ProjectFlags
  • User config is read when Flags is instantiated here. This is probably still the right place to do it, but the logic now becomes:
    • read_project_flags:
      • tries to load a user config from profiles.yml. If found, read it and raise a deprecation warning
      • if not found, tries to load user config from dbt_project.yml (can be a minimal yaml parse for just the 'flags' key - no jinja parsing necessary)
      • otherwise, use default user config settings (Nones are appropriate to ensure flag precedence works as expected for flags that can be set via CLI or env var)
  • this will also need to be updated in core.dbt.flags.set_from_args

Testing:

  • let's add a unit test that ensures that: every attribute of ProjectFlags that has a corresponding click option in params.py should be set to None by default (except for anon user tracking). Going forward, flags can have non-None defaults if they do not have a corresponding CLI option/env var. These will be used to control backwards incompatible interface or behaviour changes.

Impact to Other Teams

N/A

Will backports be required?

no :)

Context

Implications of deprecation

Why was this called UserConfig before? Well... hypothetically, if Michelle wants colorized logs, and Jeremy doesn't, they should be able to set user-specific preferences that apply across all their projects, without checking those preferences into those projects' version control (because profiles.yml lives in the HOME directory on each machine). While we are removing UserConfig from profiles.yml, it will remain possible to set user-specific preferences by setting environment variables (in .bashrc, .zshrc, etc).

@jtcohen6 jtcohen6 added the user docs [docs.getdbt.com] Needs better documentation label Nov 30, 2023
@github-actions github-actions bot changed the title UserConfig in profiles.yml → flags in dbt_project.yml [CT-3438] UserConfig in profiles.yml → flags in dbt_project.yml Nov 30, 2023
@jtcohen6 jtcohen6 changed the title [CT-3438] UserConfig in profiles.yml → flags in dbt_project.yml [CT-3438] UserConfig in profiles.yml -> flags in dbt_project.yml Nov 30, 2023
@martynydbt martynydbt added the enhancement New feature or request label Nov 30, 2023
@jtcohen6
Copy link
Contributor Author

From chatting with @gshank: Let's see what the complexity will be for implementation a "soft" deprecation. If it's a lot of additional work, we should instead opt for a "hard" deprecation in v1.8 (with a lot of prior notice), i.e. stop reading config from profiles.yml entirely.

@vianmora
Copy link

vianmora commented Feb 5, 2024

Hello :)
I'm comming from this Issue : #6207 (comment)
and before, this one : #8026

I'm trying to understand, are you going to allow the use of .env (or dbt.env) files? Or is the project abandoned as the issue is closed ? :)

@jtcohen6
Copy link
Contributor Author

jtcohen6 commented Feb 5, 2024

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

  • moving flags (f.k.a. "user configs") from profiles.ymldbt_project.yml
  • defining the intended use of flags for managing behavior changes

A few questions for you:

  • Are you still interested in using .env files to set environment variables?
  • Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
  • How are you developing/deploying dbt? (using dbt Cloud or otherwise)

@nikohunt
Copy link

Hey @vianmora! Thanks for following up here. We kicked that part out of scope for now, to refocus on:

  • moving flags (f.k.a. "user configs") from profiles.ymldbt_project.yml
  • defining the intended use of flags for managing behavior changes

A few questions for you:

  • Are you still interested in using .env files to set environment variables?
  • Are these env vars for global/runtime configs (things like DBT_FAIL_FAST), or for env vars that modify behavior in your project (accessed via {{ env_var(...) }})?
  • How are you developing/deploying dbt? (using dbt Cloud or otherwise)

@jtcohen6
I'm still interested in better handling of .env., please.

  • The env vars are for holding profile config (e.g., host, username etc.) and so not checked in to a repo.
  • I'm using dbt Core, orchestrated by Dagster.

@conorpp
Copy link

conorpp commented Apr 12, 2024

It's be great to support loading variables from .env as this is more consistent with other projects (e.g. meltano, dagster). If we can put secrets in .env then we don't need to also duplicate them to dbt's profiles.yml.

@nikohunt
Copy link

nikohunt commented Apr 12, 2024

@conorpp, I've found that adding the prefix DBT_ or DBT_ENV_SECRET_ to my variables in .env seem to cause these .env files to be used by dbt, negating the need for using set -a; source .env; set +a.

@elyobo
Copy link

elyobo commented May 3, 2024

Would this possibly have caused this issue in sqlfluff (that has definitely been caused by something in 1.7.14)? sqlfluff/sqlfluff#5841

@martanthony
Copy link

For dbt-core v1.8, we should see if the user has defined config in profiles.yml.

...

Will backports be required?

no :)

Curious to understand why this was also released to 1.6 and 1.7? There's nothing on this ticket or the pull request to indicate this should have affected versions other than 1.8

@jtcohen6
Copy link
Contributor Author

jtcohen6 commented May 3, 2024

@elyobo Thanks for flagging - I've responded on the linked sqlfluff issue.

@martanthony You're right, it wasn't our initial intent to backport this change. We decided to do this recently in order to provide existing users with the deprecation warning and mechanism to opt into more-secure (less-surprising) behavior around how built-in materializations from packages are overridden:

Apologies for not reflecting that on this issue in advance. Has it caused any issues for you, besides the one noted above for sqlfluff?

@martanthony
Copy link

@jtcohen6 understood – thanks for the prompt response. We have a step in our CI that consumes the output of dbt ls and the deprecation warning appears to have broken the way that output is handled by us. Easily mitgated by moving the profile configs to dbt_project flags though.

@polmonso
Copy link

polmonso commented Aug 7, 2024

Doesn't this change cause that everytime we start a new dbt project it will have telemetry enabled by default if we forget to turn it off?

Is the environment variable DBT_SEND_ANONYMOUS_USAGE_STATS=False the way to go for globally disabling telemetry?

@bmarcj
Copy link

bmarcj commented Sep 5, 2024

@conorpp, I've found that adding the prefix DBT_ or DBT_ENV_SECRET_ to my variables in .env seem to cause these .env files to be used by dbt, negating the need for using set -a; source .env; set +a.

This comment suggests some sort of support for '.env' files now exists in dbt. Is this actually the case (given the lack of documentation and further comment here)?

@elyobo
Copy link

elyobo commented Sep 5, 2024

@bmarcj I don't think it's built in, but I use direnv to set the variables and then you can reference them in various places in dbt.

@bmarcj
Copy link

bmarcj commented Sep 6, 2024

Thanks @elyobo! I'm looking for a minimally invasive solution for my team, so a shell extension might be out the question.

@elyobo
Copy link

elyobo commented Sep 6, 2024

I couldn't imagine life without direnv anyway, I use it for all projects. The layout options make things so much less painful as well as the env vars (no more npx <command> or poetry run dbt run or whatever). Makes me wonder why apps bother including support for e.g. .env files at all any more :D

@BI-MarcB
Copy link

BI-MarcB commented Sep 23, 2024

We currently use

config:
  log_format: json

in the profiles.yml of your production dbt user(s). This way, our devs can enjoy text-logs, while we get json output for Airflow and Datadog. Now that flags in the profile are deprecated, I am struggling to re-implement such behaviour. If I add

flags:
  log_format: "json"

to dbt_project.yml, the console output for devs is also in json. Using jinja in the flags section does not seem to be supported. My numerous tries of variation of the following did not work
log_format: '{% if target.name == "local_dev" %}json{% else %}json{% endif %}'
The above always defaults to text, even if I put json as both parts of the if-else-construct.

How can I display logs in dev console in text, but on production in json? The only "solution" that I can think of is manually adjusting every single of our 50+ dbt calls in the Airflow DAGs by adding "--log-format json" which is cumbersome, error-prone and harder to maintain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request flags user docs [docs.getdbt.com] Needs better documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.