compute: unable to read metadata from GKE

[thecodejunkie]

Client

Metadata

Environment

Google Kubernetes Engine

Expected behavior

Be able to retrieve an ID token from the metadata server when running inside GKE

Actual behavior

When following the code example (Go) for Cloud Run service-to-service authentication at https://cloud.google.com/run/docs/authenticating/service-to-service I keep getting an error when trying to retrieve an ID token when running inside GKE. I.e my caller is living in GKE and it needs to call a Cloud Run hosted service. When I try and retrieve the ID token I get the following error

2020/08/13 11:21:55 metadata: GCE metadata "/instance/service-accounts/default/identity?audience=https://[REDACTED]" not defined

However, if I manually ssh into the Kubernetes node and curl (by directly calling http://metadata.google.internal) the metadata server then I can get the ID token

[codyoss]

Thanks for the report, I will investigate and get back to you.

[codyoss]

I tried a few different things, but they all seemed to work for me.

• Used the metadata client, like the link provided.
• Used the idtoken package.
• Tried to make a request from my go program directly, with metadata.google.internal as the base host.
• Tried to make a request from my go program directly, with 169.254.169.254 as the base host.

Could you please provide me the following so I can better assist you here:

• What version is your cluster? For example I was using 1.15.12-gke.2.
• What version of cloud.google.com/go are you pulling in?
• Are you by chance setting the following environment variable: GCE_METADATA_HOST ?

[thecodejunkie]

What version is your cluster? For example I was using 1.15.12-gke.2.

The master is running on 1.15.12-gke.2 and the node pool is on 1.15.11-gke.15

What version of cloud.google.com/go are you pulling in?

Right now we have 0.59.0 and I noticed that 0.64.0 is available. I will try and update the dependency and see if that resolves it

Are you by chance setting the following environment variable: GCE_METADATA_HOST ?

No

[thecodejunkie]

We seem to be getting the same behavior with 0.64.0. The only config we have enabled on the cluster is workload identity, but that shouldn't be interfering. The cluster (dev environment) is using the default node pool service account. The deployment uses a non-default service account to run the pod, but that shouldn't be an issue either.

Also, we're running Go 1.14 inside a docker image based of scratch. Nothing out of the ordinary. This is the code we're running to try and obtain the ID token

package foo

import (
	"context"
	"fmt"

	"cloud.google.com/go/compute/metadata"
)

// MetadataIdentityProvider - defines the functionality for retreiving an identity token from
// the Google Computer Engine metadata service.
type MetadataIdentityProvider struct {
}

// NewMetadataIdentityProvider - creates a new NewMetadataIdentityProvider instance.
func NewMetadataIdentityProvider() *MetadataIdentityProvider {
	return &MetadataIdentityProvider{}
}

// GetToken - gets the token for the specified endpoint.
func (p *MetadataIdentityProvider) GetToken(ctx context.Context, endpoint string) (string, error) {
	url := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", endpoint)

	token, err := metadata.Get(url)
	if err != nil {
		return "", err
	}

	return token, nil
}

We do pass in a valid value for the endpoint parameter as I can see it being displayed in the error message (which I redacted in my initial post)

[codyoss]

Thanks for all of the extra info and providing a code example. Looking at your code I believe if you change the following line it should work.

url := fmt.Sprintf("instance/service-accounts/default/identity?audience=%s", endpoint)

Removed the leading /

[codyoss]

Note, in the next release a leading slash will be removed so this function is less error prone.

[thecodejunkie]

@codyoss I can confirm that removing the leading / did, indeed, solve it for us. Thank you!

[codyoss]

Glad I could help 😄