Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement get_column_schema_from_query #551

Merged
merged 13 commits into from
Mar 3, 2023

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Feb 22, 2023

resolves #529

Description

Checklist

satisfying 🎩 of struct types

SELECT 
1 as a,
'abc' as b,
[0, 1, 1, 2, 3, 5] AS some_numbers,
STRUCT("Rudisha" AS name, [23.4, 26.3, 26.4, 26.1] AS laps) as some_struct
models:
  - name: test_schema
    config:
      constraints:
        enabled: true
    columns:
      - name: a
        data_type: INT
        tests:
          - is_even
      - name: b
        data_type: STRING
      - name: some_numbers
        data_type: STRING
      - name: some_struct
        data_type: STRING
❯ dbt --select test_schema --project-dir ~/src/jaffle_shop
20:47:39  Running with dbt=1.5.0-a1
20:47:40  Found 6 models, 19 tests, 0 snapshots, 1 analysis, 549 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 1 metric
20:47:40  
20:47:42  Concurrency: 1 threads (target='dev')
20:47:42  
20:47:42  1 of 1 START sql table model dbt_marky.test_schematest_v2 ...................... [RUN]
20:47:43  1 of 1 ERROR creating sql table model dbt_marky.test_schematest_v2 ............. [ERROR in 1.27s]
20:47:43  
20:47:43  Finished running 1 table model in 0 hours 0 minutes and 2.60 seconds (2.60s).
20:47:43  
20:47:43  Completed with 1 error and 0 warnings:
20:47:43  
20:47:43  Compilation Error in model test_schema (models/test_schema.sql)
20:47:43    Please ensure the name, data_type, order, and number of columns in your `yml` file match the columns in your SQL file.
20:47:43    Schema File Columns: a INT64, b STRING, some_numbers STRING, some_struct STRING
20:47:43    
20:47:43    SQL File Columns: a INT64, b STRING, some_numbers ARRAY<INT64>, some_struct STRUCT<name STRING, laps ARRAY<FLOAT64>> 

fixing some_struct in schema.yml:

      - name: some_struct
        data_type: STRUCT<name STRING, laps ARRAY<FLOAT64>>
❯ dbt run --select test_schema --project-dir ~/src/jaffle_shop
20:48:49  Running with dbt=1.5.0-a1
20:48:50  Found 6 models, 19 tests, 0 snapshots, 1 analysis, 549 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 1 metric
20:48:50  
20:48:52  Concurrency: 1 threads (target='dev')
20:48:52  
20:48:52  1 of 1 START sql table model dbt_marky.test_schematest_v2 ...................... [RUN]
20:48:55  1 of 1 OK created sql table model dbt_marky.test_schematest_v2 ................. [CREATE TABLE (1.0 rows, 0 processed) in 3.47s]
20:48:55  
20:48:55  Finished running 1 table model in 0 hours 0 minutes and 4.66 seconds (4.66s).
20:48:55  
20:48:55  Completed successfully
20:48:55  
20:48:55  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

@cla-bot cla-bot bot added the cla:yes label Feb 22, 2023
@github-actions
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide.

dev-requirements.txt Outdated Show resolved Hide resolved
@MichelleArk MichelleArk marked this pull request as ready for review February 27, 2023 23:03
@MichelleArk MichelleArk requested a review from a team as a code owner February 27, 2023 23:03
mikealfare
mikealfare previously approved these changes Feb 28, 2023
@mikealfare mikealfare dismissed their stale review March 2, 2023 00:11

I have a question on the implementation in dbt-core which would flow through here.

@MichelleArk MichelleArk force-pushed the CT-2102/get_column_schema_from_query branch from 092fe2e to 25da84d Compare March 2, 2023 18:09
@MichelleArk MichelleArk closed this Mar 3, 2023
@MichelleArk MichelleArk reopened this Mar 3, 2023
@MichelleArk MichelleArk merged commit 5214ebc into main Mar 3, 2023
@MichelleArk MichelleArk deleted the CT-2102/get_column_schema_from_query branch March 3, 2023 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-2102] Implement get_column_schema_from_query macro
2 participants