Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source for specific tables #51

Merged
merged 7 commits into from
Jun 22, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
# dbt-codegen v0.4.1

## Features
- Add optional `table_names` arg to `generate_source` (#50 @rahulj51)



# dbt-codegen v0.4.0

## Breaking changes
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ which you can then paste into a schema file.
* `schema_name` (required): The schema name that contains your source data
* `database_name` (optional, default=target.database): The database that your
source data is in.
* `table_names` (optional, default=None): A list of tables that you want to generate the source definitions for.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
* `generate_columns` (optional, default=False): Whether you want to add the
column names to your source definition.
* `include_descriptions` (optional, default=False): Whether you want to add
Expand All @@ -43,7 +44,7 @@ or

```
# for multiple arguments, use the dict syntax
$ dbt run-operation generate_source --args '{"schema_name": "jaffle_shop", "database_name": "raw"}'
$ dbt run-operation generate_source --args '{"schema_name": "jaffle_shop", "database_name": "raw", "table_names":"[table_1, table_2]"}'
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
```

2. The YAML for the source will be logged to the command line
Expand Down
28 changes: 28 additions & 0 deletions integration_tests/tests/test_generate_source_some_tables.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{% set raw_schema = generate_schema_name('raw_data') %}

-- test all args
{% set actual_source_yaml = codegen.generate_source(
schema_name=raw_schema,
database_name=target.database,
table_names=['data__a_relation'],
generate_columns=True,
include_descriptions=True
) %}


{% set expected_source_yaml %}
version: 2

sources:
- name: {{ raw_schema | trim }}
tables:
- name: data__a_relation
columns:
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- name: col_a
description: ""
- name: col_b
description: ""

{% endset %}

{{ assert_equal (actual_source_yaml | trim, expected_source_yaml | trim) }}
6 changes: 5 additions & 1 deletion macros/generate_source.sql
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@


---
{% macro generate_source(schema_name, database_name=target.database, generate_columns=False, include_descriptions=False) %}
{% macro generate_source(schema_name, database_name=target.database, table_names=None, generate_columns=False, include_descriptions=False) %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sliding to the end of the signature so that it isn't a breaking change for anyone using positional arguments (rather than keyword arguments):

Suggested change
{% macro generate_source(schema_name, database_name=target.database, table_names=None, generate_columns=False, include_descriptions=False) %}
{% macro generate_source(schema_name, database_name=target.database, generate_columns=False, include_descriptions=False, table_names=none) %}


{% set sources_yaml=[] %}

Expand All @@ -29,7 +29,11 @@

{% do sources_yaml.append(' tables:') %}

{% if table_names is none %}
{% set tables=codegen.get_tables_in_schema(schema_name, database_name) %}
{% else %}
{% set tables=table_names %}
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
{% endif %}

{% for table in tables %}
{% do sources_yaml.append(' - name: ' ~ table | lower ) %}
Expand Down