-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Attempt to skip saved query processing when no semantic manifest changes #10784
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
kind: Fixes | ||
body: Attempt to skip saved query processing when no semantic manifest changes | ||
time: 2024-09-26T10:12:20.193453-04:00 | ||
custom: | ||
Author: gshank | ||
Issue: "10563" |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1141,6 +1141,24 @@ | |
|
||
def process_saved_queries(self, config: RuntimeConfig): | ||
"""Processes SavedQuery nodes to populate their `depends_on`.""" | ||
# Note: This will also capture various nodes which have been re-parsed | ||
# because they refer to some other changed node, so there will be | ||
# false positives. Ideally we would compare actual changes. | ||
semantic_manifest_changed = False | ||
SemanticManifestNode = Union[SavedQuery, SemanticModel, Metric] | ||
semantic_manifest_nodes: chain[SemanticManifestNode] = chain( | ||
self.manifest.saved_queries.values(), | ||
self.manifest.semantic_models.values(), | ||
self.manifest.metrics.values(), | ||
) | ||
for node in semantic_manifest_nodes: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: This can be simplified to
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The logic of "any" is backward, we want to execute the process metrics for node code instead of returning, but doing it that way would make merging into dbt-mantle more muddy because it executes a different function, so I'm going to leave this as it is. |
||
# Check if this node has been modified in this parsing run | ||
if node.created_at > self.started_at: | ||
semantic_manifest_changed = True | ||
break # as soon as we run into one changed node we can stop | ||
if semantic_manifest_changed is False: | ||
return | ||
|
||
current_project = config.project_name | ||
for saved_query in self.manifest.saved_queries.values(): | ||
# TODO: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Can we put this definition in a constants file somewhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved it to core/dbt/contracts/graph/nodes.py