Attempt to skip saved query processing when no semantic manifest changes #10784
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #10784 +/- ##
==========================================
- Coverage 89.00% 86.31% -2.69%
==========================================
Files 181 182 +1
Lines 23126 23297 +171
==========================================
- Hits 20583 20109 -474
- Misses 2543 3188 +645
Flags with carried forward coverage won't be shown. Click here to find out more.
|
core/dbt/parser/manifest.py
Outdated
| # because they refer to some other changed node, so there will be | ||
| # false positives. Ideally we would compare actual changes. | ||
| semantic_manifest_changed = False | ||
| SemanticManifestNode = Union[SavedQuery, SemanticModel, Metric] |
There was a problem hiding this comment.
Nit: Can we put this definition in a constants file somewhere?
There was a problem hiding this comment.
Moved it to core/dbt/contracts/graph/nodes.py
| self.manifest.semantic_models.values(), | ||
| self.manifest.metrics.values(), | ||
| ) | ||
| for node in semantic_manifest_nodes: |
There was a problem hiding this comment.
Nit: This can be simplified to
if any(node.created_at > self.started_at for node in semantic_manifest_nodes):
return
There was a problem hiding this comment.
The logic of "any" is backward, we want to execute the process metrics for node code instead of returning, but doing it that way would make merging into dbt-mantle more muddy because it executes a different function, so I'm going to leave this as it is.
Resolves #10563
Problem
We run "process_saved_queries" even when nothing has changed in the semantic manifest.
Solution
Check for changes to saved queries, semantic models, and metrics before executing process saved queries. This will have false positives because of cascading re-parsing, but will enable us to skip at least some unnecessary saved query processing.
Checklist