Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy #167

github-christophe-oudar · 2022-04-24T01:11:29Z

resolves #77

Description

Add a copy_partitions to partition_by object (default to false), throw an error if incremental_strategy != 'insert_overwrite'.
If insert_overwrite & copy_partitions, we replace the MERGE part by:
- In static mode (partitions defined), we copy explicit partitions from the output in temp table
- In dynamic mode, we retrieve the partitions from the output in temp table and copy all of them

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-bigquery next" section.

colin-rogers-dbt · 2022-10-11T21:22:21Z

dbt/adapters/bigquery/impl.py

@@ -47,11 +47,16 @@ class PartitionConfig(dbtClassMixin):
    data_type: str = "date"
    granularity: str = "day"
    range: Optional[Dict[str, Any]] = None
+    time_ingestion_partitioning: bool = False
+    copy_partitions: bool = False


@github-christophe-oudar - Is there a use case for copy_partitions outside of time_ingestion_partitioning or for not using copy_partitions with time_ingestion_partitioning? Just wondering if we need another config parameter

Yes, absolutely, it's also much faster than a merge so I'm going to use it on all my models whether it's time ingestion partitioning or column type partitioning! It really fits my use case because I'm usually processing models partition per partition but it might not fit everyone (though honestly, most people in my opinion can afford a temporary loss of consistency for such a speed gain 👍 )

Is there a use case for time_ingestion_partitioning = True and copy_partitions = False ? Should we allow users to do this?

@Fleid - I know you had concerns with adding more variables for users to set but I think we need to keep these separate

Yes, here some use cases:

time_ingestion_partitioning = True and copy_partitions = False => for atomic merge for time ingestion partitioned table (useful for people that can't afford potential inconsistency on multiple partitions)

time_ingestion_partitioning = False and copy_partitions = True =>
for fast atomic insert overwrite on a single partition or fast non atomic on multiple partitions on column type partitioning (quite common at my company for small/medium tables)

time_ingestion_partitioning = False and copy_partitions = False => before the 2 PRs

time_ingestion_partitioning = True and copy_partitions = True => for fast atomic insert overwrite on a single partition or fast non atomic on multiple partitions (a common use case at my company for large tables as it's the best performing tables from what we tested - we might need to double check now)

Great context! I'm going to do a follow up PR after we merge to update our documentation with this context.

colin-rogers-dbt

Doing some local testing, one question so far but overall it seems to function well. Hope to be able to move this and #136 forward this week

Also: apologies for the extreme delay on this, still getting ramped up but we do want to prioritize getting this merged post-Coalesce

colin-rogers-dbt · 2022-10-20T15:22:09Z

CHANGELOG.md

@@ -9,6 +9,11 @@
 - Use dbt.tests.adapter.basic in tests (new test framework) ([#135](https://github.com/dbt-labs/dbt-bigquery/issues/135), [#142](https://github.com/dbt-labs/dbt-bigquery/pull/142))
 - Adding pre-commit and black formatter hooks ([#147](https://github.com/dbt-labs/dbt-bigquery/pull/147))
 - Adding pre-commit code changes ([#148](https://github.com/dbt-labs/dbt-bigquery/pull/148))
+- Add support for ingestion time partitioned table using incremental materialization ([#136](https://github.com/dbt-labs/dbt-bigquery/pull/136))


nit: looks like you manually added a changelog entry but we're using changie? https://github.com/dbt-labs/dbt-bigquery/blob/main/CONTRIBUTING.md#updating-docs

Yes I'll update, it was done before changie was introduced

github-christophe-oudar · 2022-10-20T21:15:48Z

@colin-rogers-dbt I rebased it, I think you have put the label to trigger the integration tests so that we can check there's no regression there too.

dbt/include/bigquery/macros/materializations/incremental_strategy/insert_overwrite.sql

colin-rogers-dbt · 2022-10-25T20:01:17Z

This integ test is failing due to a known issue with our testing. No action required from @Kayrnt

colin-rogers-dbt

LGTM

cla-bot bot added the cla:yes label Apr 24, 2022

Kayrnt force-pushed the copy-partitions branch 4 times, most recently from 2274a2e to 75d90db Compare April 24, 2022 14:46

github-christophe-oudar marked this pull request as ready for review April 24, 2022 23:21

Kayrnt force-pushed the copy-partitions branch 2 times, most recently from d21d712 to 4c1e485 Compare April 29, 2022 08:26

McKnight-42 requested review from McKnight-42 and jtcohen6 May 11, 2022 16:52

McKnight-42 mentioned this pull request May 16, 2022

[CT-618] hours_to_expiration neglected when used with partitions #185

Closed

github-christophe-oudar mentioned this pull request May 20, 2022

Support for ingestion time partition table on BigQuery as incremental materialization #136

Merged

4 tasks

github-christophe-oudar mentioned this pull request Jun 10, 2022

Try using SQL for create_schema #183

Merged

4 tasks

jtcohen6 added the ready_for_review Externally contributed PR has functional approval, ready for code review from Core engineering label Jul 18, 2022

Kayrnt force-pushed the copy-partitions branch from 4c1e485 to bc71d54 Compare July 27, 2022 13:00

colin-rogers-dbt reviewed Oct 11, 2022

View reviewed changes

colin-rogers-dbt reviewed Oct 20, 2022

View reviewed changes

Kayrnt force-pushed the copy-partitions branch 2 times, most recently from 56352a8 to 333865f Compare October 20, 2022 21:10

colin-rogers-dbt added ok to test test_bigquery test all and removed ok to test labels Oct 21, 2022

McKnight-42 closed this Oct 24, 2022

McKnight-42 reopened this Oct 24, 2022

colin-rogers-dbt reviewed Oct 24, 2022

View reviewed changes

dbt/include/bigquery/macros/materializations/incremental_strategy/insert_overwrite.sql Show resolved Hide resolved

Kayrnt force-pushed the copy-partitions branch from 333865f to 803429b Compare October 24, 2022 18:35

Kayrnt force-pushed the copy-partitions branch from 803429b to 5bcff7d Compare October 24, 2022 20:36

Support copy partitions option for insert overwrite mode

5e5b391

Kayrnt force-pushed the copy-partitions branch 8 times, most recently from 9ded8d9 to 35f0e87 Compare October 25, 2022 12:22

macro renames + missing declare_dbt_max_partition

0bf3706

Kayrnt force-pushed the copy-partitions branch from 35f0e87 to 0bf3706 Compare October 25, 2022 13:22

colin-rogers-dbt approved these changes Oct 25, 2022

View reviewed changes

colin-rogers-dbt merged commit a3d7054 into dbt-labs:main Oct 25, 2022

jtcohen6 mentioned this pull request Nov 21, 2022

Update incremental modeling docs for BigQuery (ingestion-based partitioning + insert_overwrite) dbt-labs/docs.getdbt.com#2426

Closed

1 task

dbeatty10 mentioned this pull request Dec 23, 2022

[CT-1725] [CT-1722] [CT-1721] [Bug] 1.4.0-b1 Bigquery Incremental Query breaks when ending in comment dbt-labs/dbt-core#6485

Closed

2 tasks

github-christophe-oudar mentioned this pull request Apr 7, 2023

Support all types of data_type using time ingestion partitioning #496

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy #167

Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy #167

github-christophe-oudar commented Apr 24, 2022 •

edited

Loading

colin-rogers-dbt Oct 11, 2022 •

edited

Loading

github-christophe-oudar Oct 19, 2022 •

edited

Loading

colin-rogers-dbt Oct 20, 2022

github-christophe-oudar Oct 20, 2022 •

edited

Loading

colin-rogers-dbt Oct 20, 2022

colin-rogers-dbt left a comment •

edited

Loading

colin-rogers-dbt Oct 20, 2022

github-christophe-oudar Oct 20, 2022

github-christophe-oudar commented Oct 20, 2022

colin-rogers-dbt commented Oct 25, 2022

colin-rogers-dbt left a comment

Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy #167

Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy #167

Conversation

github-christophe-oudar commented Apr 24, 2022 • edited Loading

Description

Checklist

colin-rogers-dbt Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

github-christophe-oudar Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

colin-rogers-dbt Oct 20, 2022

Choose a reason for hiding this comment

github-christophe-oudar Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

colin-rogers-dbt Oct 20, 2022

Choose a reason for hiding this comment

colin-rogers-dbt left a comment • edited Loading

Choose a reason for hiding this comment

colin-rogers-dbt Oct 20, 2022

Choose a reason for hiding this comment

github-christophe-oudar Oct 20, 2022

Choose a reason for hiding this comment

github-christophe-oudar commented Oct 20, 2022

colin-rogers-dbt commented Oct 25, 2022

colin-rogers-dbt left a comment

Choose a reason for hiding this comment

github-christophe-oudar commented Apr 24, 2022 •

edited

Loading

colin-rogers-dbt Oct 11, 2022 •

edited

Loading

github-christophe-oudar Oct 19, 2022 •

edited

Loading

github-christophe-oudar Oct 20, 2022 •

edited

Loading

colin-rogers-dbt left a comment •

edited

Loading