Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit Kafka Partitions KEDA operates on #3830

Closed
AndreasBergmeier6176 opened this issue Nov 8, 2022 · 0 comments
Closed

Limit Kafka Partitions KEDA operates on #3830

AndreasBergmeier6176 opened this issue Nov 8, 2022 · 0 comments
Assignees
Labels
feature-request All issues for new features that have not been committed to needs-discussion

Comments

@AndreasBergmeier6176
Copy link
Contributor

AndreasBergmeier6176 commented Nov 8, 2022

Proposal

Instead of scaling on all Partitions, provide (sparse) lists and ranges of Partitions.
A particular KEDA instance should then only listen and act upon metrics on these Partitions.

Use-Case

  1. A Test deployment which should test data semantics on live Topic (hundreds of Partitions). To ensure that costs are not getting out of hand for the test only one or a few of these Partitions should actually be tested.
  2. A Limited Kubernetes Cluster where it is not possible to spawn a consumer for every Partition. E.g. Topic has 1000 Partitions but only able to spawn 500 Consumers on that Cluster. Thus shard Topic consuming to two Kubernetes Clusters which each handling 500 Partitions.

Anything else?

No response

@AndreasBergmeier6176 AndreasBergmeier6176 added feature-request All issues for new features that have not been committed to needs-discussion labels Nov 8, 2022
@tomkerkhove tomkerkhove changed the title Limit Partitions KEDA operates on Limit Kafka Partitions KEDA operates on Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request All issues for new features that have not been committed to needs-discussion
Projects
Archived in project
Development

No branches or pull requests

2 participants