Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lightning import throughput should be balanced across all TiKVs when the table is pre-split #54886

Closed
db-will opened this issue Jul 24, 2024 · 0 comments · Fixed by #56112
Closed
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@db-will
Copy link
Contributor

db-will commented Jul 24, 2024

Enhancement

When using Lightning to import data, if pre-split is performed on the table and node rate limiting is enabled on TiKV, it cause continuous range regions to be distributed on the same store. This results in store-level hotspots during the import process, causing the import speed to fall far short of expectations. Currently, PD only determines balance and scatter regions based on the number of regions per store. Therefore, even if Lightning calls scatter regions before data import, it does not have the desired effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant