KEDA 2.9 Crashing With Datadog Scaler

[jordanfelle]

Report

Starting in KEDA 2.9 the keda-operator has a panic every ~10 minutes when using the datadog scaler. This looks to be related to both 429s and higher latency from datadog as our APAC locations are mostly impacted.

Expected Behavior

Keda should gracefully handle issues with the datadog scaler and not crash loop. This worked in <2.9 and reverting back to 2.8.x fixes the issue.

Actual Behavior

In clusters with datadog scalers and higher latency to the datadog US apis we see panics from keda causing the pods to crash.

Steps to Reproduce the Problem

1. Install KEDA 2.9.0 or 2.9.1
2. Configure Datadog scalers and hit the rate limits + have higher latency to datadog
3. Watch for panics

Logs from KEDA operator

panic: runtime error: index out of range [0] with length 0
goroutine 800 [running]:
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).refreshScaler(0xc009e87f00, {0x419e398, 0xc008c6f600}, 0x0, 0x3e?)
/workspace/pkg/scaling/cache/scalers_cache.go:247 +0x2d6
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetScaledObjectState(0xc009e87f00, {0x419e398, 0xc008c6f600}, 0xc008a73a00, 0x419e398?)
/workspace/pkg/scaling/cache/scalers_cache.go:142 +0x757
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc001ee8240, {0x419e398, 0xc008c6f600}, {0x3942e80?, 0xc008a73a00?}, {0x418b4e0, 0xc008e054e0})
/workspace/pkg/scaling/scale_handler.go:360 +0x52e
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc001ee8240, {0x419e398, 0xc008c6f600}, 0xc008c1b540, {0x3942e80, 0xc008a73a00}, {0x418b4e0, 0xc008e054e0})
/workspace/pkg/scaling/scale_handler.go:162 +0x32a
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject
/workspace/pkg/scaling/scale_handler.go:118 +0x71d

KEDA Version

2.9.1

Kubernetes Version

1.24

Platform

Amazon Web Services

Scaler Details

Datadog

Anything else?

No response