Scalability of KEDA with Kafka scaler?

[alexandery]

Is there any data on how scalable KEDA with Kafka scaler is? Meaning - how many topics and consumer groups can I safely monitor and use for scaling deployments? I suspect the scalability would be somewhat different for different scalers.

Would I be able to monitor 10,000 topics with average of 3 consumer groups per topic? This would result in 30,000 ScaledObjects.

I was unable to find out whether KEDA runs as a single instance or multiple, distributing the load between instances.

Would appreciate any insight, pointers, suggestions.

Thank you!

P.S. Apologies for "bug" label - I wasn't sure which category to post this under.

[anirudhgarg]

Currently KEDA runs in a single pod. There has been some discussion on extending KEDA to work with multiple pods or with multiple namespaces but that hasnt been implemented.

PS: Added a new tag for "clarification-functionality"

[alexandery]

@anirudhgarg Thank you.

Is it a fair assumption that any scaler runs as a component within KEDA and as such would be limited to general limitations of the technology it monitors? With Kafka client side libraries there is definitely a limit to how many partitions one process can work with comfortably (at least Confluent ones that I've used). Would these same limits apply here?

I think MS owns Kafka scaler - https://keda.sh/scalers/apache-kafka-topic/ . Since you are from MS, do you know who would be able to answer my question?

[alexandery]

bump...

[tomkerkhove]

We are open for any contributions, regardless of who owns it!

So your concern is that its not checking all the Kafka partitions or am I getting your question wrong?

[alexandery]

@tomkerkhove My concern is the capacity of Kafka scaler within KEDA - if I need to configure scaling rules for 500-1000 (or few thousands) deployments based on 1-2-5 topics each in my Kafka cluster (which would result in at least 1-2K of topics being actively monitored by KEDA) - would KEDA support that?

I don't know how exactly Kafka scaler in KEDA monitors topics but I do know that there is a very real limit on how many Kafka partitions one process can connect to - so if Kafka scaler is a single process, then it won't be able to monitor more than 2500-3000 partitions with the Kafka client libraries I'm familiar with. That is why I was wondering about the capacity aspects of Kafka scaler.

[zroubalik]

@alexandery you can set KEDA operator to watch only particular namespace -> this allows you to deploy multiple KEDA instances in the cluster. But there could be only one Metrics Server in the cluster, so it would be shared.
Unfortunately I am not aware of any data regarding your question. If you have the resources it would be really great, if you could try to perform such load on KEDA and share the results with us!

[alexandery]

@zroubalik Thanks for your reply. Sorry, was busy with something else and am getting back to KEDA now.

Could you please elaborate a little on single Metrics server and multiple KEDA instances? For a second there I thought I could deploy KEDA into a namespace, but that's clearly not the case.

I can see what I can do to simulate the scenario I was talking about. I need to know whether we can use KEDA based on its the limitations.

[zroubalik]

@alexandery sorry, I have missed your comment somehow.

You can deploy multiple KEDA Operator instances, you need to set the WATCH_NAMESPACE variable for each deployment, ie. you can have one operator running in namespace foo the second in namespace bar and each controlling a specified namespace. Just be careful about the setting, 2 or more operators can't watch the same namespace.
Unfortunately due to the current k8s api server design, you can't deploy multiple KEDA Metrics Servers (it is the source of the metrics for HPAs which are perfoming the scaling itself). So there will be just one deployment running.

The real bottleneck could be then the Metrics Server, but I don't have any specific data on this. If you are planning to do some load/performance tests evaluation, it would be great to share the results then. Interesting scenario would be to compare results between scaling of large amount of ScaledObjects by single KEDA instance and by multiple KEDA operators, to see if/how much the single Metrics Server affects/limits the scaling.

[mariusgrigoriu]

Seems like the WATCH_NAMESPACE environment variable is not being honored by the operator.

[zroubalik]

How so?

[mariusgrigoriu]

While the adapter does honor the variable, the operator does not. Set the variable for the operator to a single namespace and you'll see in the logs that it still operates on all namespaces.

Upon inspecting the operator's code, unlike the adapter, you'll see that the namespace is not set when calling NewManager. We will submit a PR soon.

[tomkerkhove]

Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.