Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix equality test with ephemeral model + explicit column set #321

Merged
merged 2 commits into from
Jan 8, 2021

Conversation

jtcohen6
Copy link
Contributor

@jtcohen6 jtcohen6 commented Jan 7, 2021

See: dbt-labs/spark-utils#7

This is a:

  • bug fix PR with no breaking changes — please ensure the base branch is master
  • new functionality — please ensure the base branch is the latest dev/ branch
  • a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

If equality is comparing an ephemeral model with the compare_columns keyword argument, we want to avoid running adapter.get_columns_in_relation(model) entirely.

Currently, it is the default/fallback arg to kwargs.get(). Instead, we should place it inside a conditional that only runs if kwargs.get(compare_columns) returns none. I was under the impression that these were effectively the same, but this change actually fixes an integration test (test_equal_column_subset) on dbt-utils + spark-utils.

Why did this only break with Spark? Because dbt-spark uses describe table [table_name] to get the columns from a given table, a command that fails if the table does not exist. Most "core" adapters query the information schema instead; if the table doesn't exist, the query returns no results, but it doesn't fail explicitly.

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
    • Spark
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md

Copy link
Contributor

@clrcrl clrcrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me!

@jtcohen6 jtcohen6 merged commit 3a55c30 into master Jan 8, 2021
@jtcohen6 jtcohen6 deleted the fix/equality-ephemeral-col-subset branch January 8, 2021 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants