Skip to content

Commit

Permalink
Add step to run dbt from pluto build GHA
Browse files Browse the repository at this point in the history
  • Loading branch information
sf-dcp committed Sep 27, 2024
1 parent f902c94 commit 046a5e5
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 2 deletions.
8 changes: 6 additions & 2 deletions .github/workflows/pluto_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
image_tag:
type: string
required: false
build_name:
build_name:
type: string
required: true
recipe_file:
Expand Down Expand Up @@ -67,7 +67,7 @@ jobs:

- name: Set recipe env vars
working-directory: ./
run: source ./bash/export_recipe_env.sh products/pluto/${{ inputs.recipe_file }}.lock.yml
run: source ./bash/export_recipe_env.sh products/pluto/${{ inputs.recipe_file }}.lock.yml

- name: Dataloading
working-directory: products/pluto
Expand All @@ -90,3 +90,7 @@ jobs:

- name: Export and Upload
run: ./06_export.sh

- name: Custom QAQC
working-directory: products/pluto
run: ./pluto_build/07_custom_qaqc.sh
27 changes: 27 additions & 0 deletions products/pluto/pluto_build/07_custom_qaqc.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash
source ./bash/config.sh
set_error_traps

echo "Setup dbt"
dbt deps
dbt debug

echo "Build seed tables"
dbt build --select config.materialized:seed --indirect-selection=cautious --full-refresh

echo "Test source tables"
dbt test --select "source:*"

echo "Build staging tables"
dbt build --select staging

echo "Build intermediate tables"
dbt build --select intermediate

echo "Build intermediate QAQC tables"
dbt run dbt run --select qaqc # need to use 'dbt run' because otherwise it fails if downstream test fails

echo "Run DE tests"
dbt test --select tag:de_check


0 comments on commit 046a5e5

Please sign in to comment.