ARM64 image for the external snapshotter?

[ajaykmis]

Hi,

I am trying to use EBS CSI driver with the AWS EKS that has ARM nodes, but I am running into issues because the image for external-snapshotter might not be arm compatible?

I see the following issue:
~/Snapchat/dev/aws-ebs-csi-driver master*
$ kubectl get pods -n kube-system | grep ebs
ebs-snapshot-controller-0 0/1 CrashLoopBackOff 4 2m17s --->>>>>>>>

$ kubectl logs ebs-snapshot-controller-0 -n kube-system
standard_init_linux.go:219: exec user process caused: exec format error

Can someone please help with this? Do we have a arm64 image for this?

[ajaykmis]

$ kubectl describe pod ebs-snapshot-controller-0 -n kube-system
Name: ebs-snapshot-controller-0
Namespace: kube-system
Priority: 0
Node: ip-192-168-24-190.us-west-1.compute.internal/192.168.24.190
Start Time: Sun, 17 Jan 2021 20:44:24 -0800
Labels: app=ebs-snapshot-controller
app.kubernetes.io/name=aws-ebs-csi-driver
controller-revision-hash=ebs-snapshot-controller-57986cfcc
statefulset.kubernetes.io/pod-name=ebs-snapshot-controller-0
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 192.168.4.151
IPs:
IP: 192.168.4.151
Controlled By: StatefulSet/ebs-snapshot-controller
Containers:
snapshot-controller:
Container ID: docker://259562b97a3b6216edd3ff3c43140df59ca9c0cbc25d6be1e178eabe3bd25a51
Image: quay.io/k8scsi/snapshot-controller:v2.1.1
Image ID: docker-pullable://quay.io/k8scsi/snapshot-controller@sha256:2bde7d7f9d73c4309d67e9d7f94df52110295ef06c9488ca76c40da32b65f05e
Port:
Host Port:
Args:
--v=5
--leader-election=false
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sun, 17 Jan 2021 20:55:20 -0800
Finished: Sun, 17 Jan 2021 20:55:20 -0800
Ready: False
Restart Count: 7
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ebs-snapshot-controller-token-l6vkw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
ebs-snapshot-controller-token-l6vkw:
Type: Secret (a volume populated by a Secret)
SecretName: ebs-snapshot-controller-token-l6vkw
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 12m default-scheduler Successfully assigned kube-system/ebs-snapshot-controller-0 to ip-192-168-24-190.us-west-1.compute.internal
Normal Pulling 12m kubelet Pulling image "quay.io/k8scsi/snapshot-controller:v2.1.1"
Normal Pulled 12m kubelet Successfully pulled image "quay.io/k8scsi/snapshot-controller:v2.1.1"
Normal Created 10m (x5 over 12m) kubelet Created container snapshot-controller
Normal Pulled 10m (x4 over 12m) kubelet Container image "quay.io/k8scsi/snapshot-controller:v2.1.1" already present on machine
Normal Started 10m (x5 over 12m) kubelet Started container snapshot-controller
Warning BackOff 2m10s (x47 over 12m) kubelet Back-off restarting failed container

[ezzoueidi]

/assign

[ajaykmis]

I found this arm64 image for snapshot-controller:
raspbernetes/csi-external-snapshotter:2.1.1

But running this results in this error:
$ kubectl logs -f ebs-snapshot-controller-0 -n kube-system
I0119 20:29:19.125177 1 main.go:87] Version: v2.1.1-0-g034c545-dirty
I0119 20:29:19.126035 1 connection.go:153] Connecting to unix:///run/csi/socket
W0119 20:29:29.126143 1 connection.go:172] Still connecting to unix:///run/csi/socket

Any ideas?

[xing-yang]

Can you try the following images? I think you are using old images. The new ones should be multi-arch.

k8s.gcr.io/sig-storage/snapshot-controller:v2.1.3
k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3

[ajaykmis]

Thanks @xing-yang ,

I will try these out, how about these images which are referenced in setup-csi-snapshotter.yaml.

• name: csi-provisioner
  75 image: quay.io/k8scsi/csi-provisioner:v1.5.0

  • name: hostpath
    101 image: quay.io/k8scsi/hostpathplugin:v1.2.0

[ajaykmis]

Hi @xing-yang

The image for csi-snapshotter doesn't seem to work for me.

• containerID: docker://7c5fd9e7dc01f9fdd2c16ef1b0dde4cc8df8fd3e0dd1679acdc1c95b7c20d4c4
  image: k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3
  imageID: docker-pullable://k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ad9ebd1db27bc64e0d9ccac3d945fa598b7e5213aa6def217a10d6a4ae6a9770
  lastState:
  terminated:
  containerID: docker://7c5fd9e7dc01f9fdd2c16ef1b0dde4cc8df8fd3e0dd1679acdc1c95b7c20d4c4
  exitCode: 1
  finishedAt: "2021-01-20T06:52:15Z"
  reason: Error
  startedAt: "2021-01-20T06:52:15Z"
  name: csi-snapshotter
  ready: false
  restartCount: 4
  started: false
  state:
  waiting:
  message: back-off 1m20s restarting failed container=csi-snapshotter pod=ebs-csi-controller-7df9dcc9c8-ffxpl_kube-system(bab332ef-0058-4fdf-9ad7-c5ac47a04737)
  reason: CrashLoopBackOff

[Kartik494]

Hi @ajaykmis This might be an issue with image when using with EBS CSI driver, but the image you mentioned above worked fine in my enviorment k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3

[ayberk]

This is indeed due to the fact that the driver is using the old image. They were able to run the driver: kubernetes-sigs/aws-ebs-csi-driver#604

I think this can be closed.

[ajaykmis]

HI @ayberk : Sorry, these images didn't work for me. I am using these right now:
image: longhornio/csi-snapshotter:v2.1.1-lh1

When I used k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.3, I was still getting exec error.

[korbin]

k8s.gcr.io/sig-storage/snapshot-controller:v2.1.3 does not appear to be released for arm64.

[ajaykmis]

yeah, that would make sense. Are there plans to release arm64 versions for 2.1.1?

[xing-yang]

v2.1.3 image should support multiarch. I'll have to investigate why it does not work.

v2.1.1 is a little old. We won't be releasing a separate image for an old release.

[xing-yang]

Does other sidecar image such as the external-provisioner work on arm64?

[ajaykmis]

both the images you pointed didn't work for me on arm64.

for the external provisioner, this is the one I used.
raspbernetes/csi-external-provisioner

[xing-yang]

So the earlier build that supports multiarch is 3.0.x and the latest 3.0.x is 3.0.3.
The changes needed in release-tools to support multiarch didn't all go into 2.1.x.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.