Skip to content

Commit

Permalink
Add optional condition to not_null_proportion test
Browse files Browse the repository at this point in the history
The dbt core `not_null` test allows a `where` parameter and the dbt
utils `expression_is_true` allows a `condition` parameter, both of which
allow the test to operate on a subset of rows.

This adds a similar functoinality to the `not_null_proportion` test,
sticking with the `condition` convention from `expression_is_true.
  • Loading branch information
elyobo committed Sep 28, 2022
1 parent 1c8996e commit 5fdeabd
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## New features
- New feature to omit the `source_column_name` column on the `union_relations` macro ([#331](https://github.com/dbt-labs/dbt-utils/issues/331), [#624](https://github.com/dbt-labs/dbt-utils/pull/624))
- New feature to allow a `condition` option to `not_null_proportion` tests ([#691](https://github.com/dbt-labs/dbt-utils/pull/691))

## Fixes
- Better handling of whitespaces in the star macro ([#651](https://github.com/dbt-labs/dbt-utils/pull/651))
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,23 @@ models:
at_least: 0.95
```

The macro accepts an optional argument `condition` that allows for asserting the proportion is not null on a subset of all records.

**Usage:**

```yaml
version: 2
models:
- name: my_model
columns:
- name: id
tests:
- dbt_utils.not_null_proportion:
at_least: 0.95
condition: "created_at > '2018-12-31'"
```

#### not_accepted_values ([source](macros/generic_tests/not_accepted_values.sql))

Asserts that there are no rows that match the given values.
Expand Down
4 changes: 4 additions & 0 deletions integration_tests/models/generic_tests/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,10 @@ seeds:
tests:
- dbt_utils.not_null_proportion:
at_least: 0.9
- dbt_utils.not_null_proportion:
at_least: 1
at_most: 1
condition: "point_9 IS NOT NULL"

models:
- name: test_recency
Expand Down
2 changes: 2 additions & 0 deletions macros/generic_tests/not_null_proportion.sql
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@
{% set column_name = kwargs.get('column_name', kwargs.get('arg')) %}
{% set at_least = kwargs.get('at_least', kwargs.get('arg')) %}
{% set at_most = kwargs.get('at_most', kwargs.get('arg', 1)) %}
{% set condition = kwargs.get('condition', kwargs.get('arg', '1=1')) %}

with validation as (
select
sum(case when {{ column_name }} is null then 0 else 1 end) / cast(count(*) as numeric) as not_null_proportion
from {{ model }}
where {{ condition }}
),
validation_errors as (
select
Expand Down

0 comments on commit 5fdeabd

Please sign in to comment.