Improve SubCompaction Partitioning #10393
Conversation
|
This should be the implementation of #1842(Request: Optimization for subcompact: Add Get Random Keys) |
siying
force-pushed
the
subcom_partition1
branch
from
aca0e91
to
8a26d0c
Compare
|
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
siying
changed the title
siying
force-pushed
the
subcom_partition1
branch
from
c50a7bb
to
05bba31
Compare
|
@siying has updated the pull request. You must reimport the pull request before landing. |
1 similar comment
|
@siying has updated the pull request. You must reimport the pull request before landing. |
|
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| iiter_unique_ptr.reset(iiter); | ||
| } | ||
|
|
||
| const uint64_t kMaxNumAnchors = uint64_t{128}; |
There was a problem hiding this comment.
Would it be better to make it proportional to the size of the file, so small files don't have to sample that much, and make sure it also works okay for huge files.
There was a problem hiding this comment.
I can add a comment. It's not easy to figure out inside the reader whether it is a smaller file or a larger file of the compaction.
|
@siying has updated the pull request. You must reimport the pull request before landing. |
|
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
siying
force-pushed
the
subcom_partition1
branch
from
828867a
to
c79d00b
Compare
|
@siying has updated the pull request. You must reimport the pull request before landing. |
|
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: Unit tests still haven't been fixed. Also need to add more tests. But I ran some simple fillrandom db_bench and the partitioning feels reasonable. Pull Request resolved: facebook#10393 Test Plan: 1. Make sure existing tests pass. This should cover some basic sub compaction logic to be correct and the partitioning result is reasonable; 2. Add a new unit test to ApproximateKeyAnchors() 3. Run some db_bench with max_subcompaction = 4 and watch the compaction is indeed partitioned evenly. Reviewed By: jay-zhuang Differential Revision: D38043783 fbshipit-source-id: 085008e0f85f9b7c5abff7800307618320efb19f
Summary:
Unit tests still haven't been fixed. Also need to add more tests. But I ran some simple fillrandom db_bench and the partitioning feels reasonable.
Test Plan: