Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: add macro get_query_results_as_single_value #696

Merged

Conversation

CR-Lough
Copy link
Contributor

@CR-Lough CR-Lough commented Oct 1, 2022

resolves #617

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Checklist

  • [ x ] This code is associated with an Issue which has been triaged and accepted for development.
  • [ x ] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • [ x ] Postgres
    • Redshift
    • [ x ] Snowflake
  • I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • [ x ] dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • [ x ] using the limit_zero() macro in place of the literal string: limit 0
    • using dbt.type_* macros instead of explicit datatypes (e.g. dbt.type_timestamp() instead of TIMESTAMP
  • I have updated the README.md (if applicable)
  • [ x ] I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md

@joellabes joellabes linked an issue Oct 3, 2022 that may be closed by this pull request
@joellabes
Copy link
Contributor

This is cool! Thanks @CR-Lough, I'll try to review this week 🤞

@joellabes joellabes changed the base branch from main to utils-v1 October 3, 2022 00:18
Copy link
Contributor

@joellabes joellabes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the initiative to move this forward @CR-Lough!

No complaints with how the code you've written works, but a few concerns about the specific implementation choices. Let me know what you think - if you've got a specific vision in mind then I'm willing to join you on that journey, but if it's there "just in case" then I think we keep it simple and only make it more complex if needed in the future.

macros/sql/get_query_results_as_single_value.sql Outdated Show resolved Hide resolved
macros/sql/get_query_results_as_single_value.sql Outdated Show resolved Hide resolved
macros/sql/get_query_results_as_single_value.sql Outdated Show resolved Hide resolved
macros/sql/get_query_results_as_single_value.sql Outdated Show resolved Hide resolved
@joellabes
Copy link
Contributor

@CR-Lough I'm hoping to cut the release candidate of utils v1 soon and would love to include this - how are you feeling about having time to have a look at the review feedback?

@CR-Lough
Copy link
Contributor Author

Hey @joellabes , got caught up in Coalesce and the week back at work. I want to finish this up too! Will Sunday evening make the cut? Or will it be too late?

@joellabes
Copy link
Contributor

I am travelling from tomorrow and back on deck next Tuesday (NZ time), so that timing works perfectly 👌 thank you!

@joellabes
Copy link
Contributor

@CR-Lough as you might have seen, I fleshed out some tests today! Pretty happy with how it's looking now, I'll get @dbeatty10 to have a second look though since I'm now too close to it

Copy link
Contributor

@dbeatty10 dbeatty10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome to see all these edge cases covered 💪

👍 Approving


But while we're here...

If I had ruby red slippers on, I'd close my eyes, click my heels, and ask for new names for both:

  • get_query_results_as_dict
  • get_query_results_as_single_value

Design aspiration:

  • Macro names nearly as terse as run_query, but still sufficiently descriptive.

I don't personally have any great ideas here, although I appreciate the brevity that the Elementary Data folks have with these:

  • result_value
  • result_row_to_dict
  • result_column_to_list

Maybe new names like these?

  • single_result or result_singleton
  • dict_result or result_dict

'cause probs won't get a ton of love for suggesting anything with monuple or monad in the name. 😂

@joellabes
Copy link
Contributor

ask for new names for both

I was writing a bigger reply on why I wasn't going to change this (mostly around adding extra thrash to an already-thrashy upgrade season), but then realised I could get on board with get_single_value() as a sibling to get_column_values(), so I'm gonna do that, and leave get_query_results_as_dict as a slight outlier.

@joellabes joellabes merged commit 0ff9ef7 into dbt-labs:utils-v1 Nov 24, 2022
@joellabes
Copy link
Contributor

Thanks for coming on the journey @CR-Lough! great contribution 🎉

@CR-Lough
Copy link
Contributor Author

Thank you for seeing it through, @joellabes ! It was an excellent learning experience!

Also thank you @dbeatty10 !

joellabes added a commit that referenced this pull request Dec 1, 2022
* add safe_divide documentation

* add safe_divide macro

* add integration test for safe_divide macro

* Merge changes from main into utils v1 (#699)

* Correct link from README to the CONTRIBUTING guide. (#687)

* fix typo (#688)

Co-authored-by: Alex Malins <22991362+alexmalins@users.noreply.github.com>

* Change `escape_single_quotes` Reference in Pivot Macro (#692)

* Update pivot.sql

* Changelog Updates

Co-authored-by: Liam O'Boyle <github@elyobo.net>
Co-authored-by: Alex Malins <github@alexmalins.com>
Co-authored-by: Alex Malins <22991362+alexmalins@users.noreply.github.com>
Co-authored-by: zachoj10 <zjosephson@gmail.com>

* Use backwards comaptible versions of timestamp macro

* moved macro and documentation to new SQL generator section

* add tests with expressions

* fix syntax errors  (#705)

* fix syntax errors

* remove whitespace in seed file

* Restore dbt. prefix for all migrated cross-db macros (#701)

* added prefix dbt. on cross db macros

* Also prefix for new macro

* Adding changelog change

* Squashed commit of the following:

commit 5eba82b
Author: Deanna Minnick <deanna.e.minnick@gmail.com>
Date:   Wed Oct 12 10:30:42 2022 -0400

    remove whitespace in seed file

commit 7a2a5e3
Author: Deanna Minnick <deanna.e.minnick@gmail.com>
Date:   Wed Oct 12 10:22:07 2022 -0400

    fix syntax errors

Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* Remove obsolete condition argument from expression_is_true (#700)

* Remove obsolete condition argument from expression_is_true

* Improve docs

* Improve docs

* Update star.sql to allow for non-quote wrapped column names (#706)

* Update star.sql

* Update star.sql

* feat: add testing to star macro 

column encased in quotes functionality

* chore: update schema.yml

* Update star.sql

* chore: update star.sql and schema.yml

* chore: update star.sql to trim blank space

* Update README.md

* Update README.md

adds example usage of star macro's quote_identifiers argument

Co-authored-by: crlough <connor.lough@pitchbook.com>

* Switch to dbt.escape_single_quotes

* Change deprecation resolution advice

* Wrap xdb warnings in if execute block

* Slugify for snowflake (#707)

* Merge main into utils-v1 (#726)

* Feature/safe divide (#697)

* add safe_divide documentation

* add safe_divide macro

* add integration test for safe_divide macro

* moved macro and documentation to new SQL generator section

Co-authored-by: Grace Goheen <graciegoheen@gmail.com>

* Revert "Feature/safe divide (#697)" (#702)

This reverts commit f368cec.

* Quick nitpicks (#718)

I was doing some studying on these and spotted some stuff. One verb conjugation and a consistency in macro description

Co-authored-by: deanna-minnick <41010575+deanna-minnick@users.noreply.github.com>
Co-authored-by: Grace Goheen <graciegoheen@gmail.com>
Co-authored-by: ian-fahey-dbt <107962364+ian-fahey-dbt@users.noreply.github.com>

* Feat: add macro get_query_results_as_single_value (#696)

* feat: add query_results_as_single_value.sql macro

* chore: update the macro definition

Current error to work through: "failed to find conversion function from unknown to text"

* chore: update test

* chore: final edits

* chore: remove extra model reference

* chore: update return() to handle BigQuery

* chore: README.md, macro updates

* feat: factoring in first review changes

* chore: updates to testing

* chore: updates tests

* chore: update test for bigquery

* chore: update cast for bigquery

* Use example with a single record in readme

* Add default value when no record found

* test when no results are found

* Rename test file

* Add test definitions

* Fix incorrect ref

* And another one

* Update test_get_query_results_as_single_value.sql

* cast strings as strings

* Put arg in right place

* Update test_get_query_results_as_single_value.sql

* switch to limit zero for BQ

* Update test_get_query_results_as_single_value.sql

* quote column name in arg

* snowflake wont let you safe cast something to itself

* warning to future readers [skip ci]

* Add singular test to check for multi row/multi column setup

* forgot to save comment [skip ci]

* Rename to get_single_value

Co-authored-by: crlough-gitkraken <loughondata@protonmail.com>
Co-authored-by: Joel Labes <joel.labes@dbtlabs.com>

* Remove rc1 requirement for utils v1

* Recency truncate date option (#731)

* WIP changing recency test

* Add tests

* cast to timestamp for bq

* forgot the curlies

* avoid lateral column aliasing

* ts not dt

* cast source as timestamp

* don't cast inside test

* cast as date instead of truncate

* Update recency.sql

* log bq events

* store pg artifacts

* int tests dir

* Correctly store artifacts

* try casting to date or datetime

* order of operations more like order of ooperations

* dt -> ts

* Do I really have to cast this?

* Revert "Do I really have to cast this?"

This reverts commit 21e2c0d.

* Output a warning when star finds no columns, not '*' (#732)

* Change star() behaviour when no columns returned

* Code review: return a * in compile mode

* README changes

* Delete xdb_deprecation_warning.sql

* Update README.md

* Remove from ToC

* Update toc

* Fix surrogate key variable example

Co-authored-by: Deanna Minnick <deanna.e.minnick@gmail.com>
Co-authored-by: Liam O'Boyle <github@elyobo.net>
Co-authored-by: Alex Malins <github@alexmalins.com>
Co-authored-by: Alex Malins <22991362+alexmalins@users.noreply.github.com>
Co-authored-by: zachoj10 <zjosephson@gmail.com>
Co-authored-by: Grace Goheen <graciegoheen@gmail.com>
Co-authored-by: deanna-minnick <41010575+deanna-minnick@users.noreply.github.com>
Co-authored-by: Simon Quvang <sikri19@student.sdu.dk>
Co-authored-by: miles <miles@bung.cc>
Co-authored-by: Connor <61797492+CR-Lough@users.noreply.github.com>
Co-authored-by: crlough <connor.lough@pitchbook.com>
Co-authored-by: fivetran-catfritz <111930712+fivetran-catfritz@users.noreply.github.com>
Co-authored-by: ian-fahey-dbt <107962364+ian-fahey-dbt@users.noreply.github.com>
Co-authored-by: crlough-gitkraken <loughondata@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New macro: get_query_results_as_single_value
3 participants