Datadog scaler multi query

[dgibbard-cisco]

Add support for Datadog Metric API responses which container more than one Series (ie. when submitting multiple queries, comma-seperated), and allow for aggregation of results.
This allows for scaling on multiple datadog queries to be combined into a single API query, reducing Rate Limiting errors.

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated and is aligned with our changelog requirements

Fixes #

Relates to #

[dgibbard-cisco]

Have been testing this locally, added debug all over the place to track the values and it works; ie. the values are retrieved from Datadog, and aggregated as desired... but the HPA reports:

unable to get external metric dlp-scanner/s0-datadog-per_second(sum-http-requests/\u0026LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}:
unable to fetch metrics from external metrics API: no matching metrics found

the ScaledObject status is:

Status:
    Conditions:
        Message:  ScaledObject is defined correctly and is ready for scaling
        Reason:   ScaledObjectReady
        Status:   True
        Type:     Ready
        Message:  Scaling is performed because triggers are active
        Reason:   ScalerActive
        Status:   True
        Type:     Active
        Message:  No fallbacks are active on this scaled object
        Reason:   NoFallbackFound
        Status:   False
        Type:     Fallback
    External Metric Names:
        s0-datadog-per_second(sum-http-requests
    Health:
        s0-datadog-per_second(sum-http-requests:
            Number Of Failures:  35
            Status:              Failing

Events only shows:

Normal   ScaledObjectReady  5m35s (x4 over 8m52s)  keda-operator  ScaledObject is ready for scaling 

No errors...

[dgibbard-cisco]

Ah found it:

keda_metrics_adapter/datadog_scaler "msg"="error getting metrics from Datadog" "error"="query returned more than 1 series; modify the query to return only 1 series"

Probably helps when i'm using the correct image for testing the keda metrics server too 🤦