Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulk-cdk-toolkits-extract-cdc,source-mysql-v2: flesh out CDC #45887

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

postamar
Copy link
Contributor

@postamar postamar commented Sep 25, 2024

What

This PR fleshes out the Bulk CDK's CDC toolkit some more. Unit tests are missing. The level of polish here is lower than usual.

Fixes
https://github.com/airbytehq/airbyte-internal-issues/issues/9966
https://github.com/airbytehq/airbyte-internal-issues/issues/9893
https://github.com/airbytehq/airbyte-internal-issues/issues/9996

How

The main contributions here are:

  1. Moved all of what the connector needs to implement into a DebeziumOperations interface. Implementations don't need to maintain any mutable state at all.
  2. Added WAL position upper bound management in the CDK, used to terminate the feed.
  3. Ported a lot of the source-mysql CDC-related code to MySqlDebeziumOperations

Review guide

The first commit just does class and file renames. The second commit has the actual changes. There's a lot of them. They're all in the WIP areas of the codebase.

User Impact

None

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

Copy link

vercel bot commented Sep 25, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Sep 26, 2024 11:15pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/mysql-v2 labels Sep 25, 2024
@postamar postamar force-pushed the postamar/flesh-out-cdc branch 3 times, most recently from 7c76565 to 67073d3 Compare September 25, 2024 20:04
@postamar postamar changed the base branch from master to postamar/bulk-cdk-fix-state-manager-bug September 25, 2024 20:04
@postamar postamar force-pushed the postamar/flesh-out-cdc branch 2 times, most recently from bfb4312 to fc53991 Compare September 25, 2024 20:31
@postamar postamar changed the title [WIP] bulk-cdk-toolkits-extract-cdc,source-mysql-v2: flesh out CDC bulk-cdk-toolkits-extract-cdc,source-mysql-v2: flesh out CDC Sep 25, 2024
@postamar postamar marked this pull request as ready for review September 25, 2024 20:43
@postamar postamar requested a review from a team as a code owner September 25, 2024 20:43
Base automatically changed from postamar/bulk-cdk-fix-state-manager-bug to master September 25, 2024 20:44
@postamar postamar marked this pull request as draft September 26, 2024 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues CDK Connector Development Kit connectors/source/mysql-v2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants