Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-610] Prefer SQL for create_schema + drop_schema #182

Closed
jtcohen6 opened this issue May 5, 2022 · 2 comments · Fixed by #183
Closed

[CT-610] Prefer SQL for create_schema + drop_schema #182

jtcohen6 opened this issue May 5, 2022 · 2 comments · Fixed by #183
Labels
enhancement New feature or request

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented May 5, 2022

reopening #30

Today, dbt-bigquery operates as a "base" (Python) adapter for most low-level operations. This means that standard stuff like creating schemas, dropping schemas, getting columns in relations, etc, are all actually API calls wired through the Python client.

As BigQuery adds more support for SQL, it's preferable to use it where possible, and where there are few trade-offs. Two reasons:

  • It's clearer for end users to reason about, and read in the logs
  • It moves create_schema + drop_schema into "user-space" code, allowing for custom reimplementations if desired. This is a capability users have on other databases. It's especially important for create_schema + drop_schema, since these don't pass through SQL/Jinja land at all when called as adapter methods from dbt's Python tasks, namely here

The trade-offs I can imagine:

  • Performance. Generally, BigQuery's SQL "API" is always slower than its Python client API for the equivalent operation (and occasionally more expensive)
  • Reliability. get_columns_in_relation can return a contracted SchemaField, which we can turn into dbt BigQueryColumn objects, much more easily than passing query results through agate
@github-actions github-actions bot changed the title Prefer SQL for create_schema + drop_schema [CT-610] Prefer SQL for create_schema + drop_schema May 5, 2022
@jtcohen6 jtcohen6 added the enhancement New feature or request label May 10, 2022
@boxysean
Copy link

@Victoriapm look what just got merged in! I think you were looking at this, right?

@jtcohen6
Copy link
Contributor Author

@boxysean @Victoriapm We didn't manage to sneak this in ahead of cutting v1.2.0-rc1, so this won't be in for v1.2. But it will be in for v1.3! We're planning to put out a first beta (v1.3.0-b1) around the same time as v1.2.0 final, at the end of the month.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants