Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docdb] PITR: Add persistent tracking for DDL statements #8773

Open
bmatican opened this issue Jun 4, 2021 · 3 comments
Open

[docdb] PITR: Add persistent tracking for DDL statements #8773

bmatican opened this issue Jun 4, 2021 · 3 comments
Assignees
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@bmatican
Copy link
Contributor

bmatican commented Jun 4, 2021

Jira Link: DB-1829
To start with, this can be a new type of sys_catalog entry, written to from every DDL entry point. We can use a simple time based query interface, to be able to see the last N entries / last M hours of entries, via yb-admin.

This will help for PITR, as we can then be able to rollback to right before executing a particular DDL statement:

  • For YCQL, the timestamp of the actual DDL record, written from CatalogManager, might be good enough.
  • For YSQL, we might want to integrate better with the PG side of tracking, to record the start and end of a DDL transaction.
@bmatican bmatican added the area/docdb YugabyteDB core features label Jun 4, 2021
@bmatican
Copy link
Contributor Author

bmatican commented Jun 4, 2021

After the initial proof of concept, we should augment this with

  • enhance to track more types of DDLs, eg: create table is not tracked atm
  • client side filtering, eg: which record types, or which namespaces / tables are relevant, what time window
  • pagination, eg: LIMIT will be super important, to not overload the server on a request OR the client based on the RPC size, but at that point, we might as well have limit and offset
  • some debug interface, aside from just a yb-admin command, eg: either a master UI page, or RPCs that YW can call and build a UI on top of

cc @sanketkedia

spolitov added a commit that referenced this issue Jun 5, 2021
Summary:
It is a frequent case then we would like to restore the DB state to some time right before DDL operation.
For instance before the drop table.

This diff adds a DDL log, which could be seen using yb-admin.
So user could easily pick up time for restore snapshot schedule operation using this log.

Also changed UI to render time using local timezone instead of UTC.

Test Plan: ybd --cxx-test yb-admin-test --gtest_filter AdminCliTest.DdlLog

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D11744
spolitov added a commit that referenced this issue Jun 17, 2021
Summary:
It is a frequent case then we would like to restore the DB state to some time right before DDL operation.
For instance before the drop table.

This diff adds a DDL log, which could be seen using yb-admin.
So user could easily pick up time for restore snapshot schedule operation using this log.

Also changed UI to render time using local timezone instead of UTC.

Original commit: D11744 / a4ea9e4

Test Plan:
ybd --cxx-test yb-admin-test --gtest_filter AdminCliTest.DdlLog
Jenkins: rebase: 2.6

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D11952
@sanketkedia
Copy link
Contributor

I am taking point on this. cc @spolitov to avoid duplication.

@sanketkedia
Copy link
Contributor

Screen Shot 2021-07-06 at 5 58 59 AM

Screen Shot 2021-07-06 at 5 58 46 AM

Screen Shot 2021-07-06 at 5 58 33 AM

Screen Shot 2021-07-06 at 5 58 20 AM

@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 9, 2022
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature and removed kind/bug This issue is a bug labels Jul 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Development

No branches or pull requests

5 participants