AWS Pod Identity Auth Failing

[gagemillerlob]

Gloo Edge Product

Open Source

Gloo Edge Version

v1.17.7

Kubernetes Version

v1.30

Describe the bug

I setup a pod identity association for gloo edge to authenticate with AWS. My lambda upstream never populates with any upstream lambdas and the following log is seen inside of the discovery pod:

2024/09/16 10:59:38 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil> 

I found the following issue on auto scaler with the same error message
kubernetes/autoscaler#6392

It appears that the issue was resolved when updating github.com/aws/aws-sdk-go to v1.48.7 and gloo edge currently uses v1.34.9

Expected Behavior

gloo edge pods authenticate with pod identity and can make AWS calls

Steps to reproduce the bug

I used the following terraform code to configure the pod identity

module "gloo_pod_identity" {
  source = "terraform-aws-modules/eks-pod-identity/aws"
  name = "gloo"

  attach_custom_policy = true
  policy_statements = [
    {
      sid       = "Lambda"
      effect    = "Allow"
      actions   = [
        "lambda:ListFunctions",
        "lambda:InvokeFunction",
        "lambda:GetFunction",
        "lambda:InvokeAsync"
      ]
      resources = ["*"]
    }
  ]

  associations = {
    proxy = {
      cluster_name    = local.cluster_name
      namespace       = "gloo-system"
      service_account = "gateway-proxy"
    }
    discovery = {
      cluster_name    = local.cluster_name
      namespace       = "gloo-system"
      service_account = "discovery"
    }
  }
}

And these gloo settings:

settings:
  aws:
    enableCredentialsDiscovery: true

and upstream config

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: lambdas
  namespace: gloo-system
spec:
  aws:
    region: us-east-1

Additional Environment Detail

No response

Additional Context

No response