Raspberry Pi 3 does not boot with latest kernel from 21.05

[avdv]

Describe the bug

I was following the instructions from here and downloaded an image for aarch64 following the "21.05 (Latest kernel)" link (https://hydra.nixos.org/job/nixos/release-21.05/nixos.sd_image_new_kernel.aarch64-linux)

Writing that image onto a SD card and trying to boot did show this:

Error: Did not find a cmdline Flattened Device Tree
libfdt fdt_check_header(): FDT_ERR_BADMAGIC

Also reported here by somebody else: https://superuser.com/questions/1680214/rpi4-nixos-wont-boot-error-did-not-find-a-cmdline-flattened-device-tree

Following the answer from this post https://superuser.com/a/1681190/916837 and using the image build with kernel 5.10 instead made it successfully boot.

This is on a Raspberry Pi 3 Model B.

Steps To Reproduce

Steps to reproduce the behavior:

1. download https://hydra.nixos.org/build/156164445/download/1/nixos-sd-image-21.05.3801.83667ff60a8-aarch64-linux.img.zst
2. write image to SD card: zstdcat nixos-sd-image-21.05.3801.83667ff60a8-aarch64-linux.img.zst | sudo dd of=/dev/mmcblk0 bs=1M
3. insert SD card into the Pi and try to boot

Expected behavior

Successfully boot from the SD card.

[K900]

Use the latest unstable image for now. That's #139825 that probably needs to be backported, and maybe #140552 as well.

[aflatter]

For what it's worth, I tried to get NixOS up and running on an Raspberry PI 3 B+ yesterday and wasn't successful.

After flashing the image (tried both dd and the Raspberry Pi Imager application), I got stuck on the rainbow screen and the green LED flashes repeatedly 7 times in a row. According to this page, this indicates the the kernel image fails to load.

Both of these resulted in the behaviour described:

• nixos-sd-image-21.11.335376.610d4ea2750-aarch64-linux.img
• nixos-sd-image-22.05pre345062.b889981f3fc-aarch64-linux.img

Using the official distro, the device boots up nicely.

Not sure how to proceed with the debugging, hints appreciated (though it might take a couple of days until I find time to explore them).

[avdv]

Hi @aflatter ,

I just noticed that I also cannot boot my PI with any kernel > 5.10.75:

... 5.10.90-Image
48859648 bytes read 2023 ms (23 MB/s)
Retrieving file: /boot/.../rpi-3-b.dtb
14254 bytes read in 7 ms (1.9 MB/s)
Moving Image from 0x80000 to 0x200000, end=0x3140000
ERROR: RD image overlaps OS image (OS=0x200000..0x3140000)

I suppose somehow the kernel image got too big to fit the memory region before the RD image is to be loaded...

[K900]

Might be time to do the shuffle again... CC @samueldr

[samueldr]

@K900 does 21.05 (which is EOL now) have the patch?

It's possible, @avdv, that you need to update the U-Boot install currently on your Raspberry Pi.

[avdv]

@K900 does 21.05 (which is EOL now) have the patch?

Note that I have updated to 21.11 in the meantime.

It's possible, @avdv, that you need to update the U-Boot install currently on your Raspberry Pi.

Sure, how should I go about this?!

edit:

Thank you @samueldr - I downloaded the u-boot.bin file from this Hydra job: https://hydra.nixos.org/build/165287839 and put it onto the FIRMWARE partition of the SD card replacing the u-boot-rpi3.bin file. Now I can boot kernel 5.10.90 successfully, although it shows some other errors I still need to investigate but it gets to the login eventually.

[avdv]

I am closing this issue now, since 21.05 is EOL already and using an image from 21.11, specifically from this hydra job: https://hydra.nixos.org/build/166834053 works without problems:

$ nix-shell -p pv zstd
$ curl --silent -L https://hydra.nixos.org/build/166834053/download/1/nixos-sd-image-21.11.335838.11a998f6cbb-aarch64-linux.img.zst \
  | unzstd \
  | pv --wait \
  | sudo tee  /dev/mmcblk0 > /dev/null

@aflatter I suggest to open a new issue for your problem, since I am using a Pi 3 B and yours is a 3 B+ and what we are seeing are totally different symptoms.

[cyounkins]

I downloaded the u-boot.bin file from this Hydra job: https://hydra.nixos.org/build/165287839 and put it onto the FIRMWARE partition of the SD card replacing the u-boot-rpi3.bin file.

This fixed the issue for me, thank you.

I only detected this issue when my Pi failed to boot. When I look at HDMI output it appears to be trying the NixOS generations in series until it started a TFTP server. Did the kernel update break all generations, or was my Pi quietly successfully booting an old generation until it fell off the list? I'm afraid it's the latter and that's a rather insidious problem.

[avdv]

I only detected this issue when my Pi failed to boot. When I look at HDMI output it appears to be trying the NixOS generations in series until it started a TFTP server. Did the kernel update break all generations, or was my Pi quietly successfully booting an old generation until it fell off the list? I'm afraid it's the latter and that's a rather insidious problem.

Yeah, that was what I was seeing too -- it tried all boot entries one after the other until it succeeded eventually. I would assume that it really is the latter case, and I only was lucky that the 12th generation on my Pi still had the older kernel.

[cyounkins]

Unfortunately, detecting when an old generation is booted is not as simple as nix-env --list-generations. I filed a bug and a workaround in issue #163936 .

I love that automatically booting old generations helps prevent the system from bricking itself, but there's currently no follow-through, alerting the user that something is wrong with the current generation.

Without notification of failing to boot the current generation and u-boot.bin being outside the update process, I expect this to continually crop up both for this specific kernel update and other unbootable kernel issues. This same issue has already happened before: #97064