Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Allow setting of a lookback as top level model property #10662

Closed
1 task done
Tracked by #10624
QMalcolm opened this issue Sep 4, 2024 · 1 comment · Fixed by #10594
Closed
1 task done
Tracked by #10624

[Feature] Allow setting of a lookback as top level model property #10662

QMalcolm opened this issue Sep 4, 2024 · 1 comment · Fixed by #10594
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@QMalcolm
Copy link
Contributor

QMalcolm commented Sep 4, 2024

Housekeeping

  • I am a maintainer of dbt-core

Short description

In #10624 we're building a new type of incremental model strategy, microbatch. To support that we need a concept of lookback. The lookback is essentially how far to look back in multiples of the batch_size when automatically setting the event_time_start. So if event_time_end="2024-09-04 13:44:00", batch_size="day", and lookback=4, then the event_start_time would be set to 2024-08-31 00:00:00.
If defining it in the model sql it'd look like

-- my_model.sql
{{ config(
     materialization='incremental',
     incremental_strategy='microbatch',
     event_time='my_time_field',
     batch_size='day',
     lookback=0, -- int, defaults to 0 if not specified
   )
}}
...

If defining it in the model yaml it'd look like:

models:
  - name: my_model
     config:
       event_time: my_time_field
       incremental_strategy: microbatch
       batch_size: day
       lookback: 0 # int, defaults to 0 if not specified

Acceptance criteria

  • a lookback value can be specified in a model's config
  • the lookback value should default to 0 if not specified
  • the lookback value alters the generated event_time_start by a multiple of the batch_size

Suggested Tests

  • a lookback value can be specified in a model's config (via yaml and sql)
  • the lookback value is default to 0 if not specified
  • the lookback value alters the generated event_time_start by a multiple of the batch_size

Impact to Other Teams

Cloud artifacts, new property in the config of models

Will backports be required?

No

Context

No response

@QMalcolm QMalcolm added the enhancement New feature or request label Sep 4, 2024
@QMalcolm QMalcolm changed the title Allow setting of a lookback_period to effect incremental building [Feature] Allow setting of a lookback as top level model property Sep 4, 2024
@QMalcolm QMalcolm added this to the v1.9 milestone Sep 4, 2024
@QMalcolm
Copy link
Contributor Author

Resolved by #10594

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants