Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Lack of migration path from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml #10402

Open
2 tasks done
wherrmann opened this issue Jul 3, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@wherrmann
Copy link

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Initially introduced with #9289 and released with dbt-core v1.8 (though I see from #9183 that it was also backported to 1.6 and 1.7), there isn't a viable migration path for moving from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml.

The current behavior is that if user configs ARE defined in profiles.yml and the 'flags' key is NOT defined dbt_project.yml, dbt will yield this warning:

[WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.

if user configs ARE defined in profiles.yml and the 'flags' key is ALSO defined in dbt_project.yml, dbt will yield this error:

Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

The issue is that, as mentioned in dbt's documentation, profiles.yml is often not part of the dbt project and so not version-controlled:

This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control

For an organization to make this change, then, each member of the organization would need to update their profiles.yml simultaneously with the creation of the flags key in dbt_project.yml in order to avoid a runtime error.

Expected Behavior

My expectation is that setting both user configs defined in profiles.yml and the 'flags' key in dbt_project.yml would yield a deprecation warning, with the user config defined in profiles.yml being ignored in favor of the 'flags' key in dbt_project.yml. This would allow users to opt-in to the new behavior, as intended, instead of making this, effectively, a hard deprecation for any organization with more than one user.

Steps To Reproduce

  1. Use dbt-core v1.8.x
  2. Define both a user config in profiles.yml:
config:
  use_colors: false

and a 'flags' key in dbt_project.yml

flags:
  use_colors: false
  1. Run any dbt command (say, dbt debug)
  2. See runtime error
Runtime Error
  Do not specify both 'config' in profiles.yml and 'flags' in dbt_project.yml. Using 'config' in profiles.yml is deprecated.

Relevant log output

No response

Environment

- OS: macOS-14.5-x86_64
- Python: 3.11.4
- dbt: 1.8.3

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

@wherrmann wherrmann added bug Something isn't working triage labels Jul 3, 2024
@wherrmann wherrmann changed the title [Bug] Lack of migration path from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml [Bug] Lack of migration path from user configs defined in profiles.yml to the 'flags' key in dbt_project.yml Jul 3, 2024
@dbeatty10 dbeatty10 self-assigned this Jul 9, 2024
@dbeatty10
Copy link
Contributor

@wherrmann Appreciate you noticing this and taking the time to bring this to our attention 🤩

Acceptance criteria

@jtcohen6, @dataders, and myself discussed this today. We agree with your expected behavior that:

  • setting both user configs defined in profiles.yml and the flags key in dbt_project.yml should be allowed but yield a deprecation warning
  • any individual user config defined in both profiles.yml should be ignored in favor of the one under flags key in dbt_project.yml. e.g. similar to how dictionaries for the meta config are merged

The would allow multiple users of the same dbt project to add flags: to their shared dbt_project.yml file but update their local profiles.yml on their own individual timing.

Example

profiles.yml

config:
  use_colors: false
  printer_width: 80

dbt_project.yml

flags:
  use_colors: true
  partial_parse: true

After merging values from profiles.yml and dbt_project.yml (with the latter taking precedence), we'd get the following:

  use_colors: true
  partial_parse: true
  printer_width: 80

@dbeatty10 dbeatty10 removed the triage label Jul 9, 2024
@dbeatty10 dbeatty10 removed their assignment Jul 9, 2024
@dbeatty10
Copy link
Contributor

We'd backport this into 1.8 (but not earlier since it wouldn't be relevant in earlier versions).

@mahiki
Copy link

mahiki commented Sep 20, 2024

this just bit me as well. my confusion is that the only user configs in profiles.yml are:

muh_project:
   target: dev
   outputs:
    dev:
      type: databricks
      method: http
      catalog: muh_team_catalog
      schema: dev
      threads: 8
      http_path: /sql/1.0/warehouses/5xxxxxxxxxxx
      host: muh-company.cloud.databricks.com
      token: "{{ env_var('DATABRICKS_TOKEN') }}"

What of this is the 'config' part? I don't think I can eliminate these from profiles.yml and the flags in dbt_project.yml are from the elementary package and if I turn those off it will reduce functionality.

edit:

Looks like an unrelated issue specific to databricks jobs. When databricks workflow job is improperly configured it provides a default profiles.yml file and runs the models without choosing a target.

I cant find what that generated databricks profile yaml looks like yet, but when the same project was configured to specify --project-dir and --profiles-dir the error disappeared.

dbt 1.8.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants