Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add rw_actor_splits #18746

Merged
merged 3 commits into from
Sep 27, 2024

Conversation

shanicky
Copy link
Contributor

@shanicky shanicky commented Sep 27, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This pull request adds the rw_actor_splits system table, which provides the columns actor_id, split_id, source_id, and fragment_id.

dev=> create source s1 (v1 int) with (
  connector = 'kafka',
  properties.bootstrap.server = 'localhost:29092',
  topic = 't',
  scan.startup.mode = 'earliest'
) FORMAT PLAIN ENCODE JSON;
CREATE_SOURCE
dev=> create materialized view m1 as select count(*) from s1 ;
CREATE_MATERIALIZED_VIEW
dev=> SET rw_enable_shared_source TO true;
SET_VARIABLE
dev=> create source s2 (v1 int) with (
  connector = 'kafka', properties.bootstrap.server = 'localhost:29092',
  topic = 't',
  scan.startup.mode = 'earliest'
) FORMAT PLAIN ENCODE JSON;
CREATE_SOURCE
dev=> select * from rw_actor_splits;
 actor_id | split_id | source_id | fragment_id |   fragment_type
----------+----------+-----------+-------------+-------------------
        2 | 0        |         6 |           2 | NON_SHARED_SOURCE
        3 | 1        |         6 |           2 | NON_SHARED_SOURCE
        5 | 2        |         6 |           2 | NON_SHARED_SOURCE
        7 | 0        |        10 |           3 | SHARED_SOURCE
        8 | 1        |        10 |           3 | SHARED_SOURCE
        9 | 2        |        10 |           3 | SHARED_SOURCE
(6 rows)
dev=> create materialized view m2 as select count(*) from s2 ;
CREATE_MATERIALIZED_VIEW
dev=> select * from rw_actor_splits;
 actor_id | split_id | source_id | fragment_id |     fragment_type
----------+----------+-----------+-------------+------------------------
        2 | 0        |         6 |           2 | NON_SHARED_SOURCE
        3 | 1        |         6 |           2 | NON_SHARED_SOURCE
        5 | 2        |         6 |           2 | NON_SHARED_SOURCE
        7 | 0        |        10 |           3 | SHARED_SOURCE
        8 | 1        |        10 |           3 | SHARED_SOURCE
        9 | 2        |        10 |           3 | SHARED_SOURCE
       12 | 0        |        10 |           5 | SHARED_SOURCE_BACKFILL
       13 | 1        |        10 |           5 | SHARED_SOURCE_BACKFILL
       14 | 2        |        10 |           5 | SHARED_SOURCE_BACKFILL
(9 rows)

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

…ate meta_client

Signed-off-by: Shanicky Chen <peng@risingwave-labs.com>
… add `From` impl; simplify `read_rw_actor_splits`
Copy link
Member

@xxchan xxchan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others LGTM. Thanks

Can we add some tests btw? (Perhaps we can do sth like select count(*) group by source_id to avoid nondeterministic results including actor ids)

.remove(&(actor_id as _))
.unwrap_or_default()
.into_iter()
.map(move |split| list_actor_splits_response::ActorSplit {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a column type to show whether it's source fragment or backfill fragment here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to differentiate between the backfill source and the downstream?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

@xxchan
Copy link
Member

xxchan commented Sep 27, 2024

Maybe this need a release note btw

@shanicky shanicky added this pull request to the merge queue Sep 27, 2024
Merged via the queue into main with commit adc30c0 Sep 27, 2024
38 of 39 checks passed
@shanicky shanicky deleted the 18260-feat-a-system-table-to-inspect-source-splits branch September 27, 2024 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: a system table to inspect source splits
2 participants