Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graphite Scaler doesn't properly handle null responses #2944

Closed
bpinske opened this issue Apr 28, 2022 · 0 comments · Fixed by #2945 or #2954
Closed

Graphite Scaler doesn't properly handle null responses #2944

bpinske opened this issue Apr 28, 2022 · 0 comments · Fixed by #2945 or #2954
Assignees
Labels
bug Something isn't working

Comments

@bpinske
Copy link
Contributor

bpinske commented Apr 28, 2022

Report

In some configurations, it's possible for Graphite to return null for its latest datapoint. This gets erroneously interpeted as a zero.

curl 'http://graphite' 2> /dev/null |jq
[
  {
    "target": "sumSeries(metric)",
    "tags": {
      "name": "metric",
      "aggregatedBy": "sum"
    },
    "datapoints": [
      [
        15323331,
        1651092420
      ],
      [
        null,
        1651092480
      ]
    ]
  }
] 


kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/namespace/metric?labelSelector=app=app' | jq
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "metric",
      "metricLabels": null,
      "timestamp": "2022-04-27T20:47:57Z",
      "value": "0" # <<< mis-interpreting null as zero
    }
  ]
} 

Expected Behavior

Nulls are tolerated with the most recent non-null datapoint being used instead.
Nulls should not be mis-interpeted as a zero value

Actual Behavior

Null graphite responses cause apps to erroneously scale down.

Steps to Reproduce the Problem

  1. Create a graphite scaler
  2. Allow the query to return a null value
  3. Value is considered as a zero

Logs from KEDA operator

No response

KEDA Version

No response

Kubernetes Version

No response

Platform

No response

Scaler Details

No response

Anything else?

No response

@bpinske bpinske added the bug Something isn't working label Apr 28, 2022
JorTurFer pushed a commit that referenced this issue Apr 29, 2022
* Return latest non-null graphite data point. Don't mis-interpret null as zero

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Use the changelog properly

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Add test case, fix a lint failure

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Use the changelog properly

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* I'll learn how to use a changelog one day.

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

Co-authored-by: Brandon Pinske <brandon.pinske@crowdstrike.com>
bamboo12366 pushed a commit to bamboo12366/keda that referenced this issue Apr 30, 2022
 (kedacore#2945)

* Return latest non-null graphite data point. Don't mis-interpret null as zero

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Use the changelog properly

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Add test case, fix a lint failure

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* Use the changelog properly

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

* I'll learn how to use a changelog one day.

Signed-off-by: Brandon Pinske <brandon.pinske@crowdstrike.com>

Co-authored-by: Brandon Pinske <brandon.pinske@crowdstrike.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
1 participant