AMR64 images missing from ECR registry

[t0rr3sp3dr0]

/kind bug

What happened?
AMR64 images for some components are missing from the 602401143452 ECR registry.

What you expected to happen?
All AMD64 images should have an equivalent AMR64 image, just like they do on GCR.

How to reproduce it (as minimally and precisely as possible)?

$ skopeo --override-os linux --override-arch arm64 inspect docker://602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-provisioner:v2.1.1
FATA[0002] Error parsing manifest for image: choosing image instance: no image found in manifest list for architecture arm64, variant "", OS linux 

$ skopeo --override-os linux --override-arch arm64 inspect docker://602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/livenessprobe:v2.2.0
FATA[0002] Error parsing manifest for image: choosing image instance: no image found in manifest list for architecture arm64, variant "", OS linux 

$ skopeo --override-os linux --override-arch arm64 inspect docker://602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar:v2.1.0
FATA[0002] Error parsing manifest for image: choosing image instance: no image found in manifest list for architecture arm64, variant "", OS linux 

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): v1.21.2-eks-06eac09
• Driver version: v1.5.0

[t0rr3sp3dr0]

@wongma7, would you be able to check that?

[wongma7]

yes, it looks like these were built only for x64. Since the tags are immutable on EKS side, either:

• eks will need to build variants of these tags with eks suffix (most "eks add-on" images follow the convention ".eksbuild.x" https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html, the ecr public repos have another convention like "-eks-1-19-11") and then our ecr template needs to be updated
• or eks needs to build newer sidecar versions (e.g. provisioner v2.1.2) and our template needs to point to them.

I slightly prefer the second option because we are due to update sidecars across the board anyway...

[wongma7]

For the "broken" version v2.1.1, v2.2.0, and v2.1.0, the respective versions+1 v2.1.2, v2.3.0, and v2.2.0 must exist in ECR and ECR public before our template can update to them. Let me work on getting these pushed.

(Ideally ECR and ECR public would simply be mirrors of GCR i.e. some routine would basically run skopeo sync from GCR to ECR/ECR public, but for security reasons they images must be rebuilds of the code)

multi-arch image	in ECR?	in ECR public?
csi-provisioner v2.1.2	no	no
csi-provisioner v3.0.0	yes	no
livenessprobe v2.3.0	yes	no
livenessprobe v2.4.0	yes	no
node-driver-registrar v2.2.0	yes*, as of dec 9	no

[vaskozl]

This is still broken and current documentation does nothing to suggest that deployment will fail on all graviton instances.

How hard can it be to mirror a few images?

[dvaldivia]

@wongma7 so is this something that needs fixing on the ECR side to host multi-architecture images?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[t0rr3sp3dr0]

/remove-lifecycle stale

[dvaldivia]

Having support for arm64 images allows us to use graviton arm64 instances on AWS EKS which is highly desirable due to the cost savings these instances bring to the table.

[vaskozl]

@dvaldivia you don't need to sell Graviton to AWS

Did this not get fixed in a new version of the CSI driver? I thought it wasn't easily fixable due to immutable tags.