Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use reference keys instead of relations #4410

Merged
merged 5 commits into from
Dec 2, 2021
Merged

Conversation

nathaniel-may
Copy link
Contributor

Description

our Relation types override the behavior of deepcopy which means they cannot go through the json serialization pipeline which leverages asdict. This is because the dict_factory that asdict takes is not applied till the end of processing when deepcopy is internally used. The way we've overridden the behavior raises an exception when we try to serialize these values as part of a dataclass. Using reference keys instead is easier to serialize since it's just a named tuple.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change

@cla-bot cla-bot bot added the cla:yes label Dec 2, 2021
@nathaniel-may nathaniel-may marked this pull request as draft December 2, 2021 22:17
@nathaniel-may
Copy link
Contributor Author

integration tests won't really hit all the changes here. I'm going to make sure they pass locally with DBT_LOG_FORMAT=json make integration before merging.

Copy link
Contributor

@leahwicz leahwicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gshank
Copy link
Contributor

gshank commented Dec 2, 2021

It looks like the ListRelations event takes a list of relations, so we need to run the make_key thing on each element of the list

@gshank gshank marked this pull request as ready for review December 2, 2021 23:02
@@ -2499,8 +2491,8 @@ def message(self) -> str:
SQLQueryStatus(status="", elapsed=0.1)
SQLCommit(conn_name="")
ColTypeChange(orig_type="", new_type="", table="")
SchemaCreation(relation=BaseRelation())
SchemaDrop(relation=BaseRelation())
SchemaCreation(relation=_make_key(BaseRelation()))
Copy link
Contributor

@gshank gshank Dec 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have expected these lines to be: SchemaCreation(relation=_ReferenceKey). Is there some reason to call the _make_key function here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just shorter. this code is never run, just evaluated by mypy. happy to change it though if you think it's better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fine for now :)

@nathaniel-may
Copy link
Contributor Author

it's failing integration tests with json formatted logs locally so I'll be pushing a few more changes to fix those issues.

@nathaniel-may
Copy link
Contributor Author

nathaniel-may commented Dec 2, 2021

now it hangs here when DBT_LOG_FORMAT=json

you can trigger this locally with DBT_LOG_FORMAT=json python -m pytest -s test/integration/073_default_selectors_tests/test_default_selectors.py::TestDefaultSelectors::test__postgres__source_a_only

@nathaniel-may
Copy link
Contributor Author

I'm going to merge anyway to get these fixes in. I can open a new pr to fix this hang once I can nail it down.

@nathaniel-may nathaniel-may merged commit 9bdf5fe into main Dec 2, 2021
@nathaniel-may nathaniel-may deleted the nate/asdict-fixes branch December 2, 2021 23:35
leahwicz pushed a commit that referenced this pull request Dec 2, 2021
leahwicz added a commit that referenced this pull request Dec 2, 2021
Co-authored-by: Nathaniel May <nathaniel.may@fishtownanalytics.com>
iknox-fa pushed a commit that referenced this pull request Feb 8, 2022
automatic commit by git-black, original commits:
  9bdf5fe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants