Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake comments with 0.20-rc2 create a query per comment #3541

Closed
1 of 5 tasks
joshuataylor opened this issue Jul 7, 2021 · 3 comments · Fixed by #3543
Closed
1 of 5 tasks

Snowflake comments with 0.20-rc2 create a query per comment #3541

joshuataylor opened this issue Jul 7, 2021 · 3 comments · Fixed by #3543
Labels
enhancement New feature or request snowflake

Comments

@joshuataylor
Copy link
Contributor

joshuataylor commented Jul 7, 2021

Describe the bug

With 0.19, having comments creates a query like this after the table is built:

alter table foo.bar.baz alter   
        some_table COMMENT $$My super awesome comment$$ ,
        something_else COMMENT $$Hello there$$ ,

Now it does a query per comment, which drastically reduces the speed of building large tables:

comment if exists on column foo.bar.baz.some_table is $$My super awesome comment$$;

This is because of #3149 , I believe this a good fix but can lead to long build times for CI environments.

Steps To Reproduce

  • Install 0.20.0-rc2
  • Add a table with comments with Snowflake
  • Have multiple queries

Expected behavior

A single alter table.

System information

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

installed version: 0.20.0-rc2
   latest version: 0.19.2

Your version of dbt is ahead of the latest release!

Plugins:
  - snowflake: 0.20.0rc2

The output of python --version:
Python 3.8.6

@joshuataylor joshuataylor added bug Something isn't working triage labels Jul 7, 2021
@joshuataylor
Copy link
Contributor Author

This is not a bug.

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jul 7, 2021

@joshuataylor You're right, the fix in #3149 will slow down column-level persist_docs for tables with many many columns. My sense at the time is that it's worth trading the reduction in speed for the reduction in errors.

Based on what you're seeing anecdotally, how much slower is drastically slower? We could think about reworking this again for v0.21, by checking column metadata, verifying that the user-specified columns exist, and then running a single alter statement.

@jtcohen6 jtcohen6 added snowflake and removed triage labels Jul 7, 2021
@jtcohen6 jtcohen6 reopened this Jul 7, 2021
@jtcohen6 jtcohen6 added enhancement New feature or request and removed bug Something isn't working labels Jul 7, 2021
@joshuataylor
Copy link
Contributor Author

joshuataylor commented Jul 8, 2021

This is awesome, thanks so much, will try it out. It was taking about 2 extra minutes per large table (~100ms I think to run on sf, so take the round trip time maybe an extra 2s end to end?), which is painful when you are charged for the larger warehouse sizes with SF. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request snowflake
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants