-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
topicctl new action: rebalance #142
topicctl new action: rebalance #142
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think something should also be added to the rebalance section of the readme to emphasize that rebalance
is a new verb in addition to apply
for this functionality, and briefly explain how it gathers a list of topics based on yaml files found in the specified location.
README.md
Outdated
topicctl rebalance [flags] | ||
``` | ||
|
||
The `rebalance` subcommand will do a full broker rebalance for all the topic configs in a given topic prefix path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm assuming you just used the wording from the rebalance section but "full broker rebalance" suggests something else to me. How about "full cluster rebalance" or "all broker rebalance"? This change should be made in all places it appears.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"full cluster rebalance" this makes sense to me..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure about modifying at other places. But I am modifying atleast in the subcommand rebalance
Introduction:
To rebalance a topic in topicctl, We currently use topicctl apply $topic --rebalance / topicctl apply $topic --rebalance --to-remove . In a production scale Kafka cluster, there is a need to rebalance all topics for a cluster.
Goal:
Considerations for rebalance:
NOTE: A topic is not considered for rebalance if
Test Output:
Tested on local kafka cluster (3 brokers, 1 in each rack) with 3 topics
test-1 -> exists in kafka. But settings mismatch with test-1 topic yaml config
test-2 -> exists in kafka. settings match (replication 2)
test-3 -> does not exist in kafka
test-4 -> exists in kafka. settings match (replication 1)
go run cmd/topicctl/main.go -h
go run cmd/topicctl/main.go rebalance -h
go run cmd/topicctl/main.go rebalance
go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640
go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640 --to-remove 3
go run cmd/topicctl/main.go rebalance --cluster-config /path/to/cluster/cluster.yaml --path-prefix /path/to/cluster/topics/ --show-progress-interval 5s --sleep-loop-duration 15s --partition-batch-size 5 --broker-throttle-mb 640 --to-remove 2,3