[CT-3268] [Bug] dbt deps fails if a var is not defined

[darist]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

$ dbt deps
22:54:56  Running with dbt=1.6.6
22:54:56  Encountered an error:
Compilation Error
  Could not render dqm_{{ var('my_dataset') }}: Required var 'my_dataset' not found in config:
  Vars supplied to <Configuration> = {}

Where my_dataset is a var used in my dbt_project.yml without a default value, eg:

models:
  foo:
    +dataset: "{{ var('my_dataset') }}"

Expected Behavior

Expected dbt deps to succeed in bringing the dependencies to the dbt_packages folder even if I didn't pass any --vars

Based on the description at https://docs.getdbt.com/reference/commands/deps, I did not expect it to try to parse my model configs in my dbt_project.yml.

Steps To Reproduce

see current behavior

Relevant log output

No response

Environment

- OS: Linux
- Python: 3.10
- dbt: 1.6

Which database adapter are you using with dbt?

bigquery

Additional Context

This is similar to: #2368, but looks like the fix for it focused on the profiles.yml file.

Why does it matter?

We build docker images of our dbt projects to run in production. We want the deps already pulled in in the image so that we can just dbt run --vars ... in production. The values of these vars are not known at docker-image build time. We still want no default values for the vars so that a dbt run without required --vars fails early.

[dbeatty10]

Thanks for opening and including relevant links @darist !

I haven't researched why dbt_project.yml needs to be parsed for dbt deps, so I can't yet say how doable it is to remove the piece that is raising the error for you.

Have you found any workarounds?

One workaround that would be worth trying:

dbt deps --vars '{my_dataset: }'

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[mroy-seedbox]

@dbeatty10: Would love to see this reopened, as mentioned in #10829.

The workaround provided does not work in dbt Cloud as the dbt deps command is not modifiable.

And either way, installing package dependencies should have nothing to do with project variables (as opposed to running the packages later on).

Maybe packages have their own variables, but that should have nothing to do with the variables defined in my project.

Although I see now that some packages do require project-level variables/configurations. But once again, those variables should be for the execution of the package, not for its installation.

It would be nice to see a fix for this, just like it was also fixed for env vars in #2368, and for profiles in #4554.

[dbeatty10]

@mroy-seedbox re-opening per request! Thank you for your research on the other related issues 🤩

Labeling this issue as refinement for us to determine if it's a strict requirement that dbt_project.yml needs to be parsed for dbt deps or if that's something that can be bypasssed during dbt deps.

In the meantime, you could try providing a dummy default value:

models:
  foo:
    +dataset: "{{ var('my_dataset', 'ideally something that would raise an error if actually used') }}"

[mroy-seedbox]

Yeah, we are using default values as a workaround for now.

But the whole idea is to make certain variables mandatory, so ideally they should not have a default.

And since those variables as used in multiple places, raising an error is not ideal (as it would also have to be checked/copy-pasted in multiple places). 😞