-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BACKPORT 2024.1][#23367] CDCSDK: Cleanup expired and not of interest…
… tables from CDC stream Summary: ####Backport Description Major merge conflicts were encountered in cdc_service.cc in PopulateTabletCheckPointInfo() method because the refactoring present in master is not present in 2024.1 ####Original Description Currently, whenever a stream expires or a table becomes not of interest due to lack of polling on it, the entries corresponding to such tables are neither removed from the stream metadata nor are they removed from the state table. Even though the resources retained on tablets of such tables are released, the presence of state table entry can potentially stop the physical deletion of a split tablet. This diff adds cleanup mechanism for removing the entries corresponding to the expired / not of interest tables from stream metadata as well as the cdc_state table. Update Peers and Metrics reads the state table periodically and checks for the entries that have either reached stream expiry or have become not of interest. Currently this check is done for releasing the retained resources. With the changes introduced in this diff, we will find out the `{table_id, stream_id}` pairs which have expired or have become not of interest. A new rpc `RemoveTablesFromCDCSDKStream` has been introduced. This rpc takes a list of tables to be removed and the stream_id from which these tables need to be removed in its request. For each pair it then calls the `RemoveUserTablesFromCDCSDKStream` which does the cleanup from stream metadata and cdc_state table. Generally, `RemoveTablesFromCDCSDKStream` will be called for a single `{table_id, stream_id}` pair, however in case of colocated tables, all the colocated tables which have become not of interest / expired for a stream will be processed in a single call. The determination of a table being not of interest is done on the basis of active_time. In a scenario where lot of tables are present in database, but the user is interested in capturing only a small subset of these using CDC, it can happen that a large number of tables become not of interest at the same time. In such a case, there is a possibility of Update Peers and Metrics (which runs on each node) storming the master with cleanup requests. To prevent this, two throttling mechanisms have been put in place: - For each expired table, the node hosting the leader of tablet with lexicographically smallest tablet_id will send the cleanup request to master. This will ensure that only one request will be sent to the master per expired table. - If colocated tables become not of interest (or get expired for a stream), then all the colocated tables on the tablet will be cleaned up in a single call. - The flag `cdcsdk_max_expired_tables_to_clean_per_run` determines the maximum number of cleanup requests sent from each node per iteration of Update Peers and Metrics. The default value of this flag is 1. - With default settings the maximum number of cleanup requests sent to the master = **min(num of nodes, num of expired non-colocated tables + num of expired colocated tablets).** A new kLocalPersisted tserver auto flag `cdcsdk_enable_cleanup_of_expired_table_entries` has been introduced with default value false. To enable this cleanup logic, this flag must be set to true. **Upgrade/Rollback safety:** This diff introduces a new rpc: - RemoveUserTablesFromCDCSDKStream : RemoveUserTablesFromCDCSDKStreamRequestPB, RemoveUserTablesFromCDCSDKStreamResponsePB All the fields required to populate the request of this rpc are already present at the caller. The flags `cdcsdk_enable_cleanup_of_expired_table_entries` and `cdcsdk_enable_dynamic_table_addition_with_table_cleanup` protect this new rpc. Jira: DB-12291 Original commit: 0ea4f54 / D37450 Test Plan: Jenkins: urgent ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupOfTableNotOfInterest ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupOfExpiredTable ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupOfUnpolledTableWithTabletSplit ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestSplitOfTabletNotOfInterestDuringCleanup ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupOfNotOfInterestColocatedTabletWithMultipleStreams Reviewers: skumar, siddharth.shah, asrinivasan, xCluster, hsunder Reviewed By: siddharth.shah Subscribers: ycdcxcluster, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D37745
- Loading branch information
1 parent
4bcdb6c
commit 9f8c943
Showing
14 changed files
with
525 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.