Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microbatch: event_time ref + source filtering #10594

Merged
merged 41 commits into from
Sep 12, 2024
Merged

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Aug 22, 2024

Resolves #9490
Resolves #10635
Resolves #10637
Resolves #10638
Resolves #10636
Resolves #10662
Resolves #10639

Details

We're working to introduce a new feature, Microbatch Incremental Models! The goal being that microbatch incremental models will break updates to microbatch models into batched updates for a given time frame. There's more to microbatch incremental models than that, but that's the gist of if.

This PR doesn't actually do the breaking up into batches. However, what this PR does is set up the fundamentals to do so. There are a few new model configs we need: event_time, batch_size, lookback. Additionally, there are some CLI flags we need: --event-time-start and --event-time-end (with associated environment variables DBT_EVENT_TIME_END and DBT_EVENT_TIME_START). This PR makes those available and uses them to set the foundation of generating event time filters and determining when the event time filters should be generated. Finally, all this foundational functionality is gated by a new environment variable DBT_EXPERIMENTAL_MICROBATCH.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

@cla-bot cla-bot bot added the cla:yes label Aug 22, 2024
@QMalcolm QMalcolm added the Skip Changelog Skips GHA to check for changelog file label Aug 22, 2024
Copy link

codecov bot commented Aug 22, 2024

Codecov Report

Attention: Patch coverage is 96.72131% with 2 lines in your changes missing coverage. Please review.

Project coverage is 88.89%. Comparing base (ab500a9) to head (f53204f).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10594      +/-   ##
==========================================
- Coverage   88.90%   88.89%   -0.02%     
==========================================
  Files         180      180              
  Lines       22785    22842      +57     
==========================================
+ Hits        20257    20305      +48     
- Misses       2528     2537       +9     
Flag Coverage Δ
integration 86.09% <83.60%> (-0.04%) ⬇️
unit 62.44% <91.80%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Unit Tests 62.44% <91.80%> (+0.07%) ⬆️
Integration Tests 86.09% <83.60%> (-0.04%) ⬇️

@MichelleArk MichelleArk mentioned this pull request Sep 3, 2024
5 tasks
@MichelleArk MichelleArk added the artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking label Sep 3, 2024
@QMalcolm QMalcolm marked this pull request as ready for review September 12, 2024 21:40
@QMalcolm QMalcolm requested review from a team as code owners September 12, 2024 21:40
@QMalcolm QMalcolm requested review from jzhu13 and removed request for a team September 12, 2024 21:40
@MichelleArk MichelleArk requested review from gshank and removed request for jzhu13 September 12, 2024 21:40
Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self approving a PR that I was a primary contributor to 🙈

However, we've had external eyes on it. It's well gated, well tested, and confirmed backwards compatible. LGTM 🚀

@MichelleArk MichelleArk changed the title Event time ref filtering Event time ref + source filtering Sep 12, 2024
@MichelleArk
Copy link
Contributor Author

However, we've had external eyes on it. It's well gated, well tested, and confirmed backwards compatible. LGTM 🚀

echoing this as well ✅

@MichelleArk MichelleArk changed the title Event time ref + source filtering Microbatch: event_time ref + source filtering Sep 12, 2024
@MichelleArk
Copy link
Contributor Author

codecov check summaries are stale, here's what you actually see clicking into 'details':
Screenshot 2024-09-12 at 6 15 03 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment