Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microbatch Config Validation #10752

Merged

Conversation

QMalcolm
Copy link
Contributor

@QMalcolm QMalcolm commented Sep 20, 2024

Resolves #10709

Problem

We either don't validate configs for microbatch models or validate them during execution. We need parse-time validation to ensure that required fields are present and typed correctly:

  • batch_size (day, hour, month, year)
  • lookback (int)
  • event_time (str)
  • begin (datetime)

Additionally, any inputs to a microbatch model that have an event_time config must have that event_time config be a string.

Solution

  • implement parse-time validation as part of final manifest validation since we need all ref/sources resolved to scan inputs of microbatch model.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

@QMalcolm QMalcolm added the Skip Changelog Skips GHA to check for changelog file label Sep 20, 2024
@cla-bot cla-bot bot added the cla:yes label Sep 20, 2024
Copy link

codecov bot commented Sep 20, 2024

Codecov Report

Attention: Patch coverage is 91.17647% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.99%. Comparing base (730e40a) to head (c1449bc).
Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #10752   +/-   ##
=======================================
  Coverage   88.98%   88.99%           
=======================================
  Files         181      181           
  Lines       23059    23093   +34     
=======================================
+ Hits        20520    20551   +31     
- Misses       2539     2542    +3     
Flag Coverage Δ
integration 86.18% <91.17%> (+<0.01%) ⬆️
unit 62.25% <5.88%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Unit Tests 62.25% <5.88%> (-0.09%) ⬇️
Integration Tests 86.18% <91.17%> (+<0.01%) ⬆️

@MichelleArk MichelleArk removed the Skip Changelog Skips GHA to check for changelog file label Sep 24, 2024
@MichelleArk MichelleArk changed the title WIP: Microbatch Config Validation Microbatch Config Validation Sep 24, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

@MichelleArk MichelleArk marked this pull request as ready for review September 24, 2024 14:49
@MichelleArk MichelleArk requested a review from a team as a code owner September 24, 2024 14:49
Copy link
Contributor Author

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All this looks good to me 🙂

@MichelleArk MichelleArk added the artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking label Sep 24, 2024
@MichelleArk
Copy link
Contributor

schemas PR: dbt-labs/schemas.getdbt.com#58

Copy link
Contributor Author

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Looks good

@MichelleArk MichelleArk merged commit bbdb98f into main Sep 24, 2024
65 of 66 checks passed
@MichelleArk MichelleArk deleted the qmalcolm--10709-parse-time-microbatch-config-validations branch September 24, 2024 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking cla:yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement] Parse time validation of microbatch model and source configs
2 participants