pull command gets wrong architecture for recently updated official images

[morninglite]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Using a first-generation Raspberry Pi Model B, I expected to be able to run official images tagged with the linux/arm/v6 architecture, like alpine, node:alpine or debian:slim-stable.

Actual behavior

The three images listed above don't work: when attempting to start any new containers based on them, they exit immediately with no output. This happens regardless of the command specified, if any (e.g. docker run -i --rm alpine /bin/echo hello fails to produce any output), and regardless of the combination of -i -t or -d flags.

At least one image does work though: nginx:alpine, and it works using any command (e.g. docker run -i --rm nginx:alpine /bin/echo hello prints "hello") and any mode (docker run -it --rm nginx:alpine /bin/sh also works).

I couldn't discern anything obviously different between the working and non-working images. The binaries within have identical file output: "LF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, stripped".

Steps to reproduce the behavior

Simply install docker on a RaPi1B and try to run alpine or one of the other images mentioned above.

This is using the latest official RaPi image (Raspbian buster) and Docker from get.docker.com (ended up with package version 5:19.03.11~3-0~raspbian-buster from repo https://download.docker.com/linux/raspbian).

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e6
 Built:             Mon Jun  1 09:20:15 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       42e35e6
  Built:            Mon Jun  1 09:14:09 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 5
 Server Version: 19.03.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.118+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv6l
 CPUs: 1
 Total Memory: 432.4MiB
 Name: duane
 ID: BWNL:II3A:35FF:VNNQ:2QZZ:64HR:PAQF:OKYX:YDXQ:HCNL:A3JY:CLAJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpuset support

Additional environment details (AWS, VirtualBox, physical, etc.)

[morninglite]

I forgot to mention, nothing unusual is found in syslog after this happens, just the normal "shim started" and "shim reaped" from containerd and nothing from dockerd

[morninglite]

OK, so file is misleading or at least incomplete. Running readelf on the binaries revealed that the alpine image, which doesn't work, contains ARMv7 binaries and not ARMv6 as expected.

At Docker Hub, one can see that an alpine image for linux/arm/v6 exists (in addition to /v7 and others), but this is not what is being downloaded when I run docker pull alpine.

In fact, the image I get when I do that (or docker pull alpine:3.12.0) has a digest that not only doesn't match the armv6 one on the Hub, it doesn't match any of the digests listed there, for any architecture:
sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321

On the other hand, if I explicitly pull the digest the Hub lists for armv6 (docker pull alpine@sha256:71465c7d45a086a2181ce33bb47f7eaef5c233eace65704da0c5e5454a79cee5), I get the right image, and it runs as expected on the RaPi 1.

Any idea as to what is going on here? Why is it not pulling the right image automatically? And why is the image that it does pull not found on the Docker Hub at all?

Lastly, shouldn't there be an error message somewhere when exec() fails? In this case it probably returned ENOEXEC which would have made the problem a lot more obvious.

[morninglite]

OK as of today, the previously working nginx:alpine image fails too, in the same way as the others.

It was updated "a day ago" according to Docker Hub, I'm guessing this isn't a coincidence.

As with the other images, it works if you find the correct digest for the linux/arm/v6 arch on the Docker Hub website and pull that explicitly, i.e. in this case nginx@sha256:88fd834bc089b5dff1f27ab0315c3b0bbdfddfca3b2c1aeb9798944d179e441b

When just pulling nginx:alpine I instead get nginx@sha256:b89a6ccbda39576ad23fd079978c967cecc6b170db6e7ff8a769bf2259a71912.

[MC-DeltaT]

Having this problem too. I think it might not be anything to do with pull.
Using this dockerfile:

FROM scratch
ARG TARGETPLATFORM
LABEL TARGETPLATFORM=$TARGETPLATFORM

You can see that TARGETPLATFORM is linux/arm/v7 rather than linux/arm/v6. So perhaps some lower-level Docker functionality is detecting the architecture wrong?

Either way this is pretty annoying, makes it hard to do multi-architecture projects.