Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xCluster] Retain WAL upto cdc_wal_retention_time_secs if the table is part of xcluster without performing ALTER TABLE #20769

Open
yugabyte-ci opened this issue Jan 24, 2024 · 1 comment
Open

[xCluster] Retain WAL upto cdc_wal_retention_time_secs if the table is part of xcluster without performing ALTER TABLE #20769

yugabyte-ci opened this issue Jan 24, 2024 · 1 comment
Assignees
@yusong-yan
Labels
area/docdb YugabyteDB core features jira-originated kind/enhancement This is an enhancement of an existing feature priority/low Low priority

Comments

@yugabyte-ci
Copy link
Contributor

yugabyte-ci commented Jan 24, 2024 •
edited by jira bot
Loading

Jira Link: DB-9767
@yugabyte-ci yugabyte-ci added jira-originated kind/enhancement This is an enhancement of an existing feature priority/low Low priority labels Jan 24, 2024
hari90 added a commit that referenced this issue Feb 6, 2024
@hari90
[#20422] xCluster: Checkpoint table for xCluster on create
dfb067f 
Summary:
As part of creating a new table, also create xCluster stream if the table belongs to a namespace that is part of an xClusterOutboundReplicationGroup.
As part of table creation we create `AddTableToXClusterSourceTask` for each ReplicationGroup that we want to add the table to.
The task invokes `XClusterOutboundReplicationGroup::AddTable` to create a xcluster stream checkpointed to `OpId().Min()` (0,0) so that replication will start from the beginning of the WAL.
We currently skip extending WAL retention of the table since this requires an ALTER Table operation which cannot be performed unless the table creation has completed. #20769 tracks the task which will remove the need for performing ALTER Table in the first place.
#20938 tracks the race conditions when tables are created concurrently with xcluster replication groups
#16971 tracks delete of tables that are under replication

**Upgrade/Rollback safety:**
All the changes are guarded under flag `TEST_enable_xcluster_api_v2`

Fixes #20422
Jira: DB-9415

Test Plan:
TEST_F(XClusterOutboundReplicationGroupMockedTest, AddTable)
TEST_F(XClusterOutboundReplicationGroupTest, AddTable)

Reviewers: mlillibridge, jhe, slingam, xCluster

Reviewed By: mlillibridge

Subscribers: yql, xCluster, ybase, bogdan

Differential Revision: https://phorge.dev.yugabyte.com/D31935
@yugabyte-ci yugabyte-ci changed the title [xCluster] Retail WAL upto cdc_wal_retention_time_secs if the table is part of xcluster without performing ALTER TABLE [xCluster] Retain WAL upto cdc_wal_retention_time_secs if the table is part of xcluster without performing ALTER TABLE Feb 6, 2024
@yugabyte-ci yugabyte-ci assigned yusong-yan and unassigned hari90 Mar 22, 2024
@yusong-yan
Copy link
Contributor

We can adjust WAL retention time either when flag cdc_wal_retention_time_secs is updated or tablet metadata field is_under_xcluster_replication_ value changed.
However, I've identified that we are not correctly updating the is_under_xcluster_replication_ field (see Github issue: #21727). Therefore, I'll need to wait for a fix before implementing this dynamic WAL retention.

@yusong-yan yusong-yan added the area/docdb YugabyteDB core features label Mar 29, 2024
yusong-yan added a commit that referenced this issue Sep 10, 2024
@yusong-yan
[#20769] XCluster: Dynamically apply cdc_wal_retention_time_secs for …
7e40d89 
…XCluster

Summary:
Issue:
Presently we send `AlterTableRequestPB` to set the gflag`cdc_wal_retention_time_secs` on tables that are added to xCluster. Here are the limitations:
1. Removing the table from replication does not reset the value.
2. If the gflag `cdc_wal_retention_time_secs` is update, the table is not notified of it. User has to drop and recreate replication for the new value to take effect.

Improvement:
With the implementation of D36298, tablet log can determine if it is under xCluster replication by ask CDCService `CDCServiceImpl::GetXClusterMinRequiredIndex`. This allows us to dynamically use gflag `cdc_wal_retention_time_secs` when retrieving WAL retention time (`Log::wal_retention_secs`), thereby addressing the previous limitations.

New GFlag:
cdc_wal_retention_time_secs: WAL retention time in seconds to be used for tables which have a xCluster or CDCSDK outbound stream.
Jira: DB-9767

Test Plan: Jenkins

Reviewers: hsunder, xCluster

Reviewed By: hsunder

Subscribers: ybase, ycdcxcluster, jhe, mlillibridge, slingam, rthallam

Differential Revision: https://phorge.dev.yugabyte.com/D37551
jasonyb pushed a commit that referenced this issue Sep 12, 2024
@yugabyte-ci @jaki
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 98
c368a58 
Summary:
 f5ad1fb [#23855] Fixing the calculation of automatic refactoring count.
 eed826d [#22519] YSQL: Move ExplicitRowLockBuffer class into separate file
 99a27e6 [PLAT-14522] Taking yba-ctl backups with prometheus HTTPS
 e2a84b0 [PLAT-15044] Add preflight check for node addition in provider
 7e40d89 [#20769] XCluster: Dynamically apply cdc_wal_retention_time_secs for XCluster
 aa41478 [#23858] build: fix ./yb_build.sh release --gcc11
 bcf7f47 [PLAT-13910] Improve IAM credentials fetch logging and add retries
 303a202 [#23778] xCluster: Remove the capability to rename xCluster replication groups
 Excluded: 58fd26e [#23652] YSQL: Fix TestPgRegressAnalyze.java timeout / database drop failure on TSAN build
 31da65b [doc] yb_enable_bitmapscan flag (#23854)
 798db14 [#20335] DocDB: Use MonoClock for write query metric
 80779d8 [#23860] xCluster: Add automatic ddl mode proto fields
 afd763d Revert "Revert "[PLAT-14786] Add support to node_agent install to use bind ip and node_external_fqdn""
 e86951a [#23841] docdb: Disable stack trace tracking in TSAN builds
 d600608 [PLAT-15244] Fix schedule not getting updated on edit schedule API call
 3c0df09 [PLAT-15214][PLAT-15232]YBC version upgrade to 2.2.0.0-b6 and enable YBC verbose by default
 aa7372e [#23478] YSQL: fix connection manager session variable case sensitivity issue
 0d53558 [PLAT-14810][PLAT-14811][YBA CLI] Support adding and editing EIT configurations
 ffa537e [PLAT-10706][dr] Support retry-ability of failover and switchover
 Excluded: 5ae4558 [#23578] YSQL: Add HELP and TYPE to :13000/prometheus-metrics
 3aa7459 [PLAT-15016] Handle gflag_group changes for ENHANCED_PG_COMPATIBILITY group in 2024.1.3
 c89356c [PLAT-10592][YBA] Changes to support global tserver/master service in K8s
 7fc3b76 [#3893] YCQL: Align 2 system_schema.* tables with Cassandra
 da6274e [23646] Test Stability: Fix PgMiniTest.FollowerReads
 5fa6dc9 [PLAT-15180][Platform][UI][PITR]Create Restore Backup modal
 be0d1d1 [PLAT-15247][Platform][Backup]Create Backup scheduled policy List

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, tfoucher

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D37981
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yusong-yan yusong-yan

Labels
area/docdb YugabyteDB core features jira-originated kind/enhancement This is an enhancement of an existing feature priority/low Low priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hari90 @yugabyte-ci @yusong-yan