overriding kernel fails (no new deployment and "grub2-mkconfig: Child process exited with code 1")

[ganchair]

Host system details

State: idle
Warning: failed to finalize previous deployment
         error: Bootloader write config: grub2-mkconfig: Child process exited with code 1
         check `journalctl -b -1 -u ostree-finalize-staged.service`
Deployments:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220810.0 (2022-08-10T00:46:50Z)
                   Commit: 0adbf32421d4bf13c9d691b7ea8019f539f47002988e988078383413bfb73aa5
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

  fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220809.0 (2022-08-09T00:50:58Z)
                   Commit: 519760c927003a00fc2e5adbbbc81e37657e69cd4a0793507784e09daab1ce7d
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4

Expected vs actual behavior

# rpm-ostree override replace kernel-5.19.1-300.fc36.x86_64.rpm kernel-core-5.19.1-300.fc36.x86_64.rpm kernel-modules-5.19.1-300.fc36.x86_64.rpm kernel-modules-extra-5.19.1-300.fc36.x86_64.rpm

no error here but after reboot there is no new deployement

after reboot to switch the new deployment

In rpm-ostree status I get

Warning: failed to finalize previous deployment
         error: Bootloader write config: grub2-mkconfig: Child process exited with code 1
         check `journalctl -b -1 -u ostree-finalize-staged.service`

and its just booted in to the same old deployment.

And here is the output of that journalctl:

Aug 11 22:57:48 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Aug 11 22:57:58 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Aug 11 22:57:58 fedora ostree[23226]: Finalizing staged deployment
Aug 11 22:57:59 fedora ostree[23226]: Copying /etc changes: 22 modified, 0 removed, 76 added
Aug 11 22:57:59 fedora ostree[23226]: Copying /etc changes: 22 modified, 0 removed, 76 added
Aug 11 22:58:01 fedora ostree[23226]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 1
Aug 11 22:58:01 fedora systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Aug 11 22:58:01 fedora systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Aug 11 22:58:01 fedora systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Aug 11 22:58:01 fedora systemd[1]: ostree-finalize-staged.service: Consumed 1.763s CPU time.

Steps to reproduce it

1. get the kernel rpm from "https://koji.fedoraproject.org/koji/taskinfo?taskID=90704186"
2. try to override
3. reboot and see rpm-ostree status

[miabbott]

Similar looking report on /r/Fedora - https://www.reddit.com/r/Fedora/comments/wlwkxg/cant_layer_any_packages_on_silverblue/

[jlebon]

This could be #3715. Can you try the workaround in #3715 (comment)?

[jlebon]

Related: https://pagure.io/workstation-ostree-config/pull-request/288

[ganchair]

I ran find /boot/efi -exec touch '{}' ';' with sudo and then override the kernel and still get the same error. Should I do this after the override?

[ganchair]

ok. I also had to run sudo ostree admin finalize-staged worked now.

[ImmortalTux]

Having this issue as well. Running sudo ostree admin finalize-staged did not work for me.

[lukeallister]

Wanted to chime in to say I had the same issue today trying to install tailscale on a new install of kinoite. The linked fix worked for me:

sudo find /boot/efi -exec touch '{}' ';'
sudo rpm-ostree install tailscale
sudo ostree admin finalize-staged
systemctl reboot

[busla]

Confirming that #3715 (comment) fixed the issue on version: 36.20220810.0

find /boot/efi -exec touch '{}' ';'
sudo ostree admin finalize-staged
systemctl reboot

[PotatoesFall]

Running

sudo find /boot/efi -exec touch '{}' ';'

sadly did not resolve the issue for me. I ran this both before and after using rpm-ostree to install neovim.

error message in journalctl is identical to first post in issue.

@busla is there something else you did before/after running the command?

EDIT:
running

sudo ostree admin finalize-staged

after the rpm-ostree install command was the fix, just like in #3925 (comment)

[busla]

apologies 🙏

I ran that also but didnt paste the line in my comment.

Updated.

[RHOOPH]

Although it needs to be run each time an upgrade or installation is made, this worked for me. I hope this will soon be fixed.

Confirming that #3715 (comment) fixed the issue on version: 36.20220810.0

find /boot/efi -exec touch '{}' ';'
sudo ostree admin finalize-staged
systemctl reboot

[mikezila]

Similar issues are bubbling up on both Ask Fedora and reddit. The fixes proposed seem to work for some users but not others:

https://ask.fedoraproject.org/t/fedora-silverblue-36-will-not-succesfully-deploy-after-layering-packages/25352/15

[subvert0r]

Can someone ELI5 what is the root cause of this bug, and is there any security risk involved (Other than not updating your system) ?

[jlebon]

I think this is a dupe of fedora-silverblue/issue-tracker#322. Let's track it there.