Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure microbatch models respect model level full_refresh config during batch creation #10785

Closed
Tracked by #10624
QMalcolm opened this issue Sep 26, 2024 · 0 comments · Fixed by #10788
Closed
Tracked by #10624
Assignees
Milestone

Comments

@QMalcolm
Copy link
Contributor

QMalcolm commented Sep 26, 2024

In dbt-core, it is generally expected that values passed via CLI flags take precedence over model level configs. However, full_refresh on a model is an exception to this rule, where in the model config takes precedence. This config exists specifically to prevent accidental full refreshes of large incremental models, as doing so can be costly. It is actually best practice to set full_refresh=False on incremental models.

Currently, for microbatch models the above is not happening. The CLI flag --full-refresh is taking precedence over the model config full_refresh. That means that currently if --full-refresh is supplied then the microbatch model will be refreshed even if full_refresh=False is set on the model. This needs to not be the case.

@QMalcolm QMalcolm changed the title Ensure microbatch models respect model level full_refresh config Ensure microbatch models respect model level full_refresh config during batch creation Sep 26, 2024
@QMalcolm QMalcolm self-assigned this Sep 26, 2024
@QMalcolm QMalcolm added this to the v1.9 milestone Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant