Raspberry Pi 3 B+ can't boot with latest kernel 5.7.19

[tobiasBora]

Describe the bug
Since my last kernel upgrade on the Raspberry Pi 3B+, I can't boot anymore (I lost ssh access). I tried to burn a new image from scrach, update the channel, and create the most basic configuration, and the problem persists.

To Reproduce
Steps to reproduce the behavior:

1. Download the Aarch64 sdcard image here
2. Unpack, and burn it with dd on a sdcard
3. Make sure that you have the following config.txt (enable_uart is useful to boot with uart, and force_turbo was required to have less noise in the characters read by uart):

kernel=u-boot-rpi3.bin

# Boot in 64-bit mode.
arm_control=0x200

# U-Boot used to need this to work, regardless of whether UART is actually used or not.
# TODO: check when/if this can be removed.
enable_uart=1

# Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
# when attempting to show low-voltage or overtemperature warnings.
avoid_warnings=1
force_turbo=1

4. Boot on this sdcard, and I recommend to use UART + screen to boot with a serial console
5. Configure internet, put the following file in /etc/nixos/configuration.nix:

{ config, pkgs, lib, ... }:
{
  # NixOS wants to enable GRUB by default
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;
 
  # !!! Otherwise (even if you have a Raspberry Pi 2 or 3), pick this:
  boot.kernelPackages = pkgs.linuxPackages_latest;
  
  # !!! Needed for the virtual console to work on the RPi 3, as the default of 16M doesn't seem to be enough.
  # If X.org behaves weirdly (I only saw the cursor) then try increasing this to 256M.
  # On a Raspberry Pi 4 with 4 GB, you should either disable this parameter or increase to at least 64M if you want the USB ports to work.
  boot.kernelParams = ["cma=32M"];
    
  # File systems configuration for using the installer's partition layout
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
    
  # !!! Adding a swap file is optional, but strongly recommended!
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];
}

and then run:

sudo nix-channel --update
sudo nixos-rebuild switch

Reboot: the OS stops booting at Starting kernel ..., even if you add in the kernel parameters console=ttyS1,115200n8 or console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7 (via /boot/extlinux/extlinux.conf):

------------------------------------------------------------
1:      NixOS - Default
2:      NixOS - Configuration 2 (2020-09-04 00:54 - 20.03.2873.10aede9bcfe)
3:      NixOS - Configuration 1 (1970-01-01 00:00 - 20.03.2869.0d60b0b10ea)
Enter choice: 1:        NixOS - Default
Retrieving file: /boot/extlinux/../nixos/jkjjy892iirdhns5mxqx063xn374ph80-initrd-linux-5.7.19-initrd
7726240 bytes read in 322 ms (22.9 MiB/s)
Retrieving file: /boot/extlinux/../nixos/hmwv1a4bnlshi1zlnzsk83sg41ybaf73-linux-5.7.19-Image
39021056 bytes read in 1614 ms (23.1 MiB/s)
append: systemConfig=/nix/store/jrmziij585fgr7n82dg3ff435s13gnhb-nixos-system-nixos-20.03.2873.10aede9bcfe init=/nix/store/jrmziij585fgr7n82dg3ff435s13gnhb-nixos-system-nixos-20.03.2873.10aede9bcfe/init cma=32M console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
Retrieving file: /boot/extlinux/../nixos/hmwv1a4bnlshi1zlnzsk83sg41ybaf73-linux-5.7.19-dtbs/broadcom/bcm2837-rpi-3-b-plus.dtb
14622 bytes read in 6 ms (2.3 MiB/s)
## Flattened Device Tree blob at 02600000
   Booting using the fdt blob at 0x2600000
   Using Device Tree in place at 0000000002600000, end 000000000260691d

Starting kernel ...

Moreover, I think it is not a problem with UART, because on my first system I couldn't ssh my system that was working fine during 2 years, and the green LED does not blink during the expected boot period.

Expected behavior
I would expect it to boot properly.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I tried with the kernel 5.6 and it boots fine. I experienced issues with the kernel 5.7.18 and 5.7.19

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[Mic92]

I can confirm this issue. My current workaround is to use:

boot.kernelPackages = pkgs.linuxPackages_rpi3; # v4.19.118 on my system

or:

boot.kernelPackages = pkgs.linuxPackages; # at the time of writing v5.4.66

instead of:

boot.kernelPackages = pkgs.linuxPackages_latest; # at the time of writing v5.8.11

Note that hdmi does not display the boot for pkgs.linuxPackages but with pkgs.linuxPackages_rpi3, it works after a while however.

[lucernae]

Can also confirm this issue. My current workaround is to use:

boot.kernelPackages = pkgs.linuxPackages_5_4

then nixos-rebuild switch.

For anyone noobs like me, in the original issue after doing upgrade you lose access of the pi after upgrading to boot.kernelPackages = pkgs.linuxPackages_latest. You can't use keyboard and the boot are stuck to the default boot with Starting kernel... messages hanging via HDMI. SSH doesn't seems to work (maybe the pi hasn't boot at all). To recover access, I did the following:

1. My recovery pi setup consists of USB keyboard + HDMI + Ethernet. But sometimes your USB keyboard is undetected. Try to unplug and put it on other slot. Sometimes it works. When it works, immediately press any button to get into uboot. Proceed with the boot by typing boot, but then you get to choose which generation of Nixos you want to boot. Choose your last known configuration then, work from there to recover.

2. If above method fails, extract your SD card and mount it to any linux OS. However, I only have SD card reader over MacOS, but mac can't mount ext4 device. So I use live usb in my macos just to get into a linux OS. From there mount the sdcard manually:

mount -t ext4 /dev/disk/by-label/NIXOS_SD /mnt/pi

Where /mnt/pi is your mount point. From there, edit /mnt/pi/boot/extlinux/extlinux.conf and change the default boot to your last know configuration to recover.

I think for useful reason this notes can be included in the NixOS ARM wiki. It could be helpful for noobs like me out there. A better way to test new image would be to just use nixos-rebuild boot then manually choose the new generation at boot, so if it doesn't work, the boot will choose your working default setup.

[Mic92]

I fixed the wiki for the time being: https://nixos.wiki/wiki/NixOS_on_ARM#NixOS_installation_.26_configuration Maybe we should mark 5.7+ as broken on raspberry pi 4?

[rjpcasalino]

confirming that boot.kernelPackages = pkgs.linuxPackages_5_4 helped me work around this issue whereas the wiki still mentions: boot.kernelPackages = pkgs.linuxPackages;. I'm on a Pi 3. My machine would hang on
Starting kernel ...

[Mic92]

@rjpcasalino linuxPackages.kernel is currently 5.4.78. So it should be the same as pkgs.linuxPackages_5_4

[ghost]

Just a quick heads-up that I was able to boot Linux 5.10.2 on the Raspberry Pi 3 with the following patch:

File rpi.nix

{ lib, pkgs, config, modulesPath, ... }:

let
  uboot = pkgs.buildUBoot rec {
    defconfig = "rpi_3_defconfig";
    extraMeta.platforms = ["aarch64-linux"];
    filesToInstall = ["u-boot.bin"];
    version = "2021.01-rc4";
    src = pkgs.fetchurl {
      url = "ftp://ftp.denx.de/pub/u-boot/u-boot-${version}.tar.bz2";
      sha256 = "1f7qzz8nfrvi8r77h07mspl7xrc3br9rhb1njj9ylfm5cli19jg4";
    };
    patches = [ ./0001-configs-rpi-allow-for-bigger-kernels.patch ];
    enableParallelBuilding = true;
  };

in {
  imports = [
    (modulesPath + "/installer/cd-dvd/sd-image.nix")
  ];

  boot.loader.grub.enable = false;
  boot.loader.generic-extlinux-compatible.enable = true;
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.initrd.availableKernelModules = [
    # Allows early (earlier) modesetting for the Raspberry Pi
    "vc4" "bcm2835_dma" "i2c_bcm2835"
  ];
  boot.consoleLogLevel = lib.mkDefault 7;

  sdImage = {
    populateFirmwareCommands = let
      configTxt = pkgs.writeText "config.txt" ''
        kernel=u-boot-rpi.bin

        # Boot in 64-bit mode.
        arm_64bit=1

        # U-Boot used to need this to work, regardless of whether UART is actually used or not.
        # TODO: check when/if this can be removed.
        enable_uart=1

        # Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
        # when attempting to show low-voltage or overtemperature warnings.
        avoid_warnings=1
      '';
      in ''
        (cd ${pkgs.raspberrypifw}/share/raspberrypi/boot && cp bootcode.bin fixup*.dat start*.elf $NIX_BUILD_TOP/firmware/)
        cp ${uboot}/u-boot.bin firmware/u-boot-rpi.bin
        cp ${configTxt} firmware/config.txt
      '';
    populateRootCommands = ''
      mkdir -p ./files/boot
      ${config.boot.loader.generic-extlinux-compatible.populateCmd} -c ${config.system.build.toplevel} -d ./files/boot
    '';
  };

  documentation.enable = false;
  networking.hostName = "hifipi";
  networking.interfaces.eth0.useDHCP = true;

  system.stateVersion = "21.03";
}

File 0001-configs-rpi-allow-for-bigger-kernels.patch

From 3c6b7c0922370e9d0c1705706e7c47dcd234e6c8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Milan=20P=C3=A4ssler?= <milan@petabyte.dev>
Date: Wed, 30 Dec 2020 11:49:16 +0100
Subject: [PATCH] configs/rpi: allow for bigger kernels

---
 include/configs/rpi.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/configs/rpi.h b/include/configs/rpi.h
index 834f1cd2..b63ee96f 100644
--- a/include/configs/rpi.h
+++ b/include/configs/rpi.h
@@ -163,10 +163,10 @@
    "fdt_high=" FDT_HIGH "\0" \
    "initrd_high=" INITRD_HIGH "\0" \
    "kernel_addr_r=0x00080000\0" \
-   "scriptaddr=0x02400000\0" \
-   "pxefile_addr_r=0x02500000\0" \
-   "fdt_addr_r=0x02600000\0" \
-   "ramdisk_addr_r=0x02700000\0"
+   "scriptaddr=0x02c00000\0" \
+   "pxefile_addr_r=0x02d00000\0" \
+   "fdt_addr_r=0x02e00000\0" \
+   "ramdisk_addr_r=0x02f00000\0"

 #if CONFIG_IS_ENABLED(CMD_MMC)
    #define BOOT_TARGET_MMC(func) \
--
2.29.2

Build using nixpkgs be0b453 and write to sd:

nix build -f "<nixpkgs/nixos>" config.system.build.sdImage --system aarch64-linux -I nixos-config=rpi.nix -I nixpkgs=https://github.com/nixos/nixpkgs/archive/be0b453d7c7eee2090962c9a83749b024ff9acf5.tar.gz
pv result/sd-image/nixos-sd-image-21.03pre-git-aarch64-linux.img.zst | zstd -d | sudo dd of=/dev/sdX bs=1M oflag=dsync

[sdier]

I was able to boot using 'setenv' for each of the values in the last update and then using:
sysboot mmc 0:2 any 0x02c00000 /boot/extlinux/extlinux.conf

This might help someone who just wants to boot up and go back to another kernel for now.

[blaggacao]

@petabyteboy Do you plan to process your findings in a way they emerge out of this thread toward some sort of greater audience (upstreaming, PR to nixpkgs, etc.)?

[ghost]

@petabyteboy Do you plan to process your findings in a way they emerge out of this thread toward some sort of greater audience (upstreaming, PR to nixpkgs, etc.)?

• I'm not sure if patching the u-boot source code is the best way to fix this, I would think these values should be configurable somehow, but I don't know how
• we'll have to keep increasing it as kernels get bigger and bigger
• existing installations will still break when updating to newer, larger kernels as there is no boot loader installation script to update u-boot to a new version

So the patch I posted was more of a temporary solution, but it could still be helpful for many people to package it in nixpkgs until a better way is known.

[ghost]

For the third bullet point also see #108048

[blaggacao]

So the patch I posted was more of a temporary solution, but it could still be helpful for many people to package it in nixpkgs until a better way is known.

Would it hurt?

[ghost]

No, feel free to make a PR, but I would add it to the u-boot packaging instead of making it specific to the sd-image.

[blaggacao]

feel free to make a PR

Unfortunately, I don't think I can take the pass right now. So I'll mirror this to interested parties. But thanks for the discussion. I'm concluding: your patch is apt to be incorporated in principle in nixpkgs to make Rpi3 with latest kernel work out of the box until a better fix is found later.

[ghost]

#108975

[blaggacao]

Thanks! I just learned something 😄 .

[ghost]

For reference: If you are experiencing this issue on an existing installation, try to replace the files in your firmware partition with those from a recent sdImage.

[cyounkins]

@petabyteboy thank you for merging that in! Should we be looking at 20.09 release sdimage or unstable? It looks like Hydra has not yet built either that include the diff, so those looking to update their firmware partition from an sdimage may need to wait a few hours.

[cyounkins]

Looks like the patch failed to apply on Hydra: https://hydra.nixos.org/build/134735802/nixlog/10

[ghost]

Looks like the patch failed to apply on Hydra: https://hydra.nixos.org/build/134735802/nixlog/10

Yes, it's already fixed. Just waiting for the next evaluation.

[cyounkins]

I can confirm that on my RPi 3B (nonplussed I believe), the patch by @petabyteboy as built into the firmware partition of this image permits me to load kernel 5.9.16 whereas before I would have the OP symptoms of hanging at "Starting Kernel..." and no system services starting.

I am confused though. "Starting Kernel..." is output by syslinux, which is started by u-boot. At the point of issue, control has transferred to syslinux. Isn't the kernel loaded by u-boot a sort of mini-kernel that is independent of the system kernel? Or is there a leaky abstraction that I don't fully understand?

Once I better understand this, I'm happy to update the comments in rpi.h and message the u-boot mailing list. Thanks!

[ghost]

This build includes u-boot 2021.01 and the patch to support the larger mainline kernels again: https://hydra.nixos.org/build/135056994
It should boot both on the Pi 3 and Pi 4.