Support for wal2json plugin #22449
Assignees
Labels
area/cdcsdk
CDC SDK
jira-originated
kind/new-feature
This is a request for a completely new feature
priority/highest
Highest priority issue
Comments
yugabyte-ci
added
area/cdcsdk
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Jun 11, 2024
…ge transactions.' Summary: This diff imports the upstream PG commit `Fix the logical replication timeout during large transactions.` - `87c1dd246af8ace926645900f02886905b889718`. PG commit: postgres/postgres@87c1dd2 PG commit description: ``` The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com ``` Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* Reviewers: stiwary Reviewed By: stiwary Subscribers: yql, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35740
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Jun 11, 2024
Summary: This diff introduces the support for `wal2json` output plugin with the PG compatible logical replication support. Key points: # A commit from PG (postgres/postgres@87c1dd2) was needed. This will be added in a separate revision (https://phorge.dev.yugabyte.com/D35740) # Like test_decoding this plugin does not send the relation object. So the schema refresh callback is a NOOP. # The sql and expected files from the wal2json repo and the corresponding PgRegressTests have not been added since it requires support for PG functions such as `pg_logical_slot_get_changes` which is currently unsupported. They will be imported separately as part of issue: #22687 Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#testWithWal2JsonPlugin' Manually tested with pg_recvlogical Reviewers: stiwary, asrinivasan, skumar Reviewed By: stiwary, skumar Subscribers: yql, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35521
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Jun 12, 2024
…imeout during large transactions.' Summary: Original commit: cd5af9e / D35740 This diff imports the upstream PG commit `Fix the logical replication timeout during large transactions.` - `87c1dd246af8ace926645900f02886905b889718`. PG commit: postgres/postgres@87c1dd2 PG commit description: ``` The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com ``` ######Backport Description No merge conflicts were encountered. Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* Reviewers: stiwary Reviewed By: stiwary Subscribers: ycdcxcluster, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35743
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Jun 12, 2024
Summary: Original commit: b361d4e / D35521 This diff introduces the support for `wal2json` output plugin with the PG compatible logical replication support. Key points: # A commit from PG (postgres/postgres@87c1dd2) was needed. This will be added in a separate revision (https://phorge.dev.yugabyte.com/D35740) # Like test_decoding this plugin does not send the relation object. So the schema refresh callback is a NOOP. # The sql and expected files from the wal2json repo and the corresponding PgRegressTests have not been added since it requires support for PG functions such as `pg_logical_slot_get_changes` which is currently unsupported. They will be imported separately as part of issue: #22687 ######Backport Description Minor merge conflicts were encountered in `src/postgres/contrib/Makefile` Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#testWithWal2JsonPlugin' Manually tested with pg_recvlogical Reviewers: stiwary, asrinivasan, skumar Reviewed By: stiwary Subscribers: ycdcxcluster, yql Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35751
karthik-ramanathan-3006
pushed a commit
to karthik-ramanathan-3006/yugabyte-db
that referenced
this issue
Jun 24, 2024
…ring large transactions.' Summary: This diff imports the upstream PG commit `Fix the logical replication timeout during large transactions.` - `87c1dd246af8ace926645900f02886905b889718`. PG commit: postgres/postgres@87c1dd2 PG commit description: ``` The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com ``` Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* Reviewers: stiwary Reviewed By: stiwary Subscribers: yql, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35740
karthik-ramanathan-3006
pushed a commit
to karthik-ramanathan-3006/yugabyte-db
that referenced
this issue
Jun 24, 2024
Summary: This diff introduces the support for `wal2json` output plugin with the PG compatible logical replication support. Key points: # A commit from PG (postgres/postgres@87c1dd2) was needed. This will be added in a separate revision (https://phorge.dev.yugabyte.com/D35740) # Like test_decoding this plugin does not send the relation object. So the schema refresh callback is a NOOP. # The sql and expected files from the wal2json repo and the corresponding PgRegressTests have not been added since it requires support for PG functions such as `pg_logical_slot_get_changes` which is currently unsupported. They will be imported separately as part of issue: yugabyte#22687 Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#testWithWal2JsonPlugin' Manually tested with pg_recvlogical Reviewers: stiwary, asrinivasan, skumar Reviewed By: stiwary, skumar Subscribers: yql, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35521
devansh-ism
pushed a commit
that referenced
this issue
Jul 23, 2024
Summary: Subtree merge wal2json into src/postgres/third-party-extensions/wal2json. Ran the following: ``` git remote add wal2json https://github.com/eulerto/wal2json.git git fetch wal2json wal2json_2_6:tags/wal2json_2_6 git subtree add --prefix=src/postgres/third-party-extensions/wal2json wal2json wal2json_2_6 ``` ``` Add 'src/postgres/third-party-extensions/wal2json/' from commit '75629c2e1e81a12350cc9d63782fc53252185d8d' git-subtree-dir: src/postgres/third-party-extensions/wal2json git-subtree-mainline: 66ed3a5 git-subtree-split: 75629c2 ``` Choose wal2json_2_6 since it is the latest version that supports PG 11 according to their [[ https://github.com/eulerto/wal2json/tree/master?tab=readme-ov-file#unix-based-operating-systems | README ]]. Integrating wal2json into the build and YB specific changes will be done as a followup diff. Jira: DB-11366 Test Plan: jenkins: compile only Reviewers: sumukh.phalgaonkar, jason, skumar Reviewed By: jason Subscribers: steve.varnau, yql Differential Revision: https://phorge.dev.yugabyte.com/D36690
jasonyb
pushed a commit
that referenced
this issue
Jul 23, 2024
Summary: 5a76f6a [#23179] CDCSDK: Support data types with dynamically alloted oids in CDC 5820ccd [PLAT-14710] Do not return apiToken in response to getSessionInfo dcfa9cd [docs] updates to CVE table status column (#23225) c0b1ee8 [docs] Fix load balance keyword in drivers page (#23253) 2becb46 [docs] Add basic troubleshooting steps for read restart errors. (#23228) 1b9be2e [PLAT-12733] Kubernetes overrides in v2 api 50422f8 [#23011] YSQL: Enable ALTER TABLE IF EXISTS t RENAME c1 TO c2 7c55b95 [PLAT-14073] DB scoped failover+repair 5ac65eb [#22449] YSQL: import wal2json wal2json_2_6 9e046fb [#23163] YSQL: pg_partman: make 'inherit_template_properties' idempotent 2248dcd [#23163] YSQL: pg_partman: make 'apply_publications' idempotent 7b32d05 [#23163] YSQL: pg_partman: Make partition creation idempotent b0349fe [PLAT-14710] update api doc for /session_info fe37ffd [#23240] CDCSDK: Make test replicationConnectionConsumptionAllDataTypes more resilient Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36789
devansh-ism
pushed a commit
that referenced
this issue
Jul 25, 2024
Summary: Diff D35521 incorrectly imported `wal2json` plugin to contrib folder but it should have been added to third-party-extensions. It also added YB specific changes on top. So two things are required to fix this. - Add wal2json as subtree merge to third-party-extensions folder. - Add YB specific changes on top of that. First part of subtree done by diff D36690. This diff addresses the part to add YB specific changes. Difference between `third-party-extensions/wal2json` and `contrib/wal2json`: - `sql/` and `expected/` folders that were removed from contrib/wal2json are restored in third-party-extensions/wal2json. Although not added to a test schedule. - `README.md` that was from contrib/wal2json is restored in third-party-extensions/wal2json. Made inline with [[ https://github.com/eulerto/wal2json/tree/wal2json_2_6 | thirdparty ]] repo. - `wal2json.sln` and `wal2json.vcxproj` : provide build configuration to build on windows. Restored them and inline with thirdparty repo. - `logical.conf` which was present in contrib/wal2json not added in third-party-extensions/wal2json , not required. - `.gitignore` now inline with thirdparty repo. - `Makefile` now inline with thirdparty repo. No change required as plugin is built without any changes. Jira: DB-11366 Test Plan: Jenkins: test regex: .*ReplicationSlot.* ./yb_build.sh --java-test 'org.yb.pgsql.TestPgReplicationSlot#testWithWal2JsonPlugin' Reviewers: sumukh.phalgaonkar, jason, skumar Reviewed By: jason Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D36694
jasonyb
pushed a commit
that referenced
this issue
Jul 25, 2024
Summary: 5aa0c0a [PLAT-14078] Add local provider test for update databases cdd97f8 remove ea badge (#23276) 2813d78 [PLAT-14156][PLAT-14323]: Move all UI Driven flags to INTERNAL and remove YBM key as its not used 49523f5 [PLAT-14733]: Add support for OIDC attributes jwt_jwks_path and jwt_jwks_url b039d1a [PLAT-14366] Basic local provider test for master auto failover 700fd49 [#23275] docdb: Fix missing home icon on master UI 89e434e [#13254] YSQL: import pgtap v1.3.3 1b3585f [doc][cdc] Updated diagrams (#23262) 399f165 [#23266] YSQL: Only require YB Admin privileges to run pg_locks 5a4bbd4 [#19954] docdb: Register both tablet split children atomically b4c4294 [PLAT-14617] Add support for numerical search and enable extra search fields forxCluster 84fb7ad [#22449] YSQL: wal2json YB specific changes afe84d4 [#13254] YSQL: add pgtap to build adf3c54 [#23272] YSQL, ASH: Fix incorrect popping of query id from nested query ids stack 3b42c2e [docs] Add syntax documentation for logical replication (#23270) Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36841
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-11366
The text was updated successfully, but these errors were encountered: