Add Support for Mutli-Queries in Datadog Scaler

[dgibbard-cisco]

Signed-off-by: Darren Gibbard dgibbard@cisco.com

Adds support for comma-separated multi-queries in the Datadog Scaler.

Previously, if a query returned more than one element, we would throw an error. With this change, when Datadog returns multiple Series elements, we then apply an average or max operation (set by the queryAggregator parameter) on the latest points from each Series to aggregate the results to a single return value.

This is beneficial to reducing the number of Datadog API Requests being made, which can quickly hit rate limits which impacts scaling.

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated and is aligned with our changelog requirements

Fixes #3423

Doc PR Link: kedacore/keda-docs#885

[dgibbard-cisco]

Would be good to get @arapulido 's eyes on this too if available, as the main code creator :)

[arapulido]

Thanks for the contribution!

I am not super convinced about this, as it starts implementing things like avg or max outside Datadog.

I would accept the change though if we warn the user clearly, and we don't use defaults that can hide what the scaler is doing.

I have added ways of doing that in the inline review. I will review the docs as well.

[dgibbard-cisco]

I am not super convinced about this, as it starts implementing things like avg or max outside Datadog.

Some context here; Datadog doesn't support running avg/max operations across the results of multiple queries (eg; something like max(query1,query2) is not valid); but they do support returning multiple query result sets, in the form of multiple Series elements.
So we have to do the aggregation ourselves as a result; hence why I got stuck with this :D

[dgibbard-cisco]

Pushed some changes so that:

• Stop non-explicit usage of query aggregation -- if query returns multiple Series and queryAggregator is not set, throw error:

query returned more than 1 series; modify the query to return only 1 series or add a queryAggregator

• Cleanup logic and remove FindStringInSlice()

@arapulido I think this covers the desire to not have it default, and/or display an error when not explicitly set? As well as the suggested cleanup :)
We still use MaxFloatFromSlice() as the default result return method, just out of convenience of not having to split it if Series is 1 or more.

[arapulido]

LGTM, thanks!

[zroubalik]

/run-e2e datadog*
Update: You can check the progress here

[dgibbard-cisco]

Resolved conflict with #3617 :)
Might need e2e tests kicked again?

[dgibbard-cisco]

@zroubalik - can I trouble you for a re-kick for e2e tests?

[zroubalik]

/run-e2e datadog*
Update: You can check the progress here

[JorTurFer]

LGTM!
Thanks for the contribution! ❤️