zfs: Fail on boot due to zfs mount point being non-empty

[nialov]

Describe the bug

With the following fileSystems, the /home mount fails due to "non-empty
directory there" (logs further below):

{
  "/" = {
    autoFormat = false;
    autoResize = false;
    depends = [ ];
    device = "zroot/root/nixos";
    encrypted = {
      blkDev = null;
      enable = false;
      keyFile = null;
      label = null;
    };
    formatOptions = "";
    fsType = "zfs";
    label = null;
    mountPoint = "/";
    neededForBoot = false;
    noCheck = false;
    options = [ "defaults" "zfsutil" ];
  };
  "/boot" = {
    autoFormat = false;
    autoResize = false;
    depends = [ ];
    device =
      "/dev/disk/by-id/ata-Samsung_SSD_850_PRO_256GB_S1SUNSAFB04586A-part1";
    encrypted = {
      blkDev = null;
      enable = false;
      keyFile = null;
      label = null;
    };
    formatOptions = "";
    fsType = "vfat";
    label = null;
    mountPoint = "/boot";
    neededForBoot = false;
    noCheck = false;
    options = [ "defaults" ];
  };
  "/home" = {
    autoFormat = false;
    autoResize = false;
    depends = [ ];
    device = "zroot/root/home";
    encrypted = {
      blkDev = null;
      enable = false;
      keyFile = null;
      label = null;
    };
    formatOptions = "";
    fsType = "zfs";
    label = null;
    mountPoint = "/home";
    neededForBoot = false;
    noCheck = false;
    options = [ "defaults" "zfsutil" ];
  };
}

The relevant error log parts from sudo journalctl --boot is as follows
(cleaned some probably irrelevant logs from the middle):

Jan 14 09:37:46 nixos-desktop zfs[1008]: cannot mount 'zroot/root/home': mountpoint or dataset is busy
Jan 14 09:37:46 nixos-desktop systemd[1]: Mounted FUSE Control File System.
Jan 14 09:37:46 nixos-desktop systemd[1]: Mounted Kernel Configuration File System.
Jan 14 09:37:46 nixos-desktop systemd[1]: Finished Apply Kernel Variables.
Jan 14 09:37:46 nixos-desktop systemd[1]: Finished Create Static Device Nodes in /dev.
Jan 14 09:37:46 nixos-desktop systemd[1]: Reached target Preparation for Local File Systems.
Jan 14 09:37:46 nixos-desktop systemd[1]: home.mount: Directory /home to mount over is not empty, mounting anyway.
Jan 14 09:37:46 nixos-desktop systemd[1]: Mounting /home...
Jan 14 09:37:46 nixos-desktop systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jan 14 09:37:46 nixos-desktop systemd[1]: Finished Load/Save Random Seed.
Jan 14 09:37:46 nixos-desktop systemd-udevd[1080]: Using default interface naming scheme 'v252'.
Jan 14 09:37:46 nixos-desktop systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jan 14 09:37:46 nixos-desktop systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Jan 14 09:37:46 nixos-desktop systemd[1]: Failed to start Mount ZFS filesystems.

Output from zfs list is as follows:

➜ zfs list
NAME               USED  AVAIL     REFER  MOUNTPOINT
zroot             19.9G   211G       24K  /zroot
zroot/root        19.9G   211G       25K  /zroot/root
zroot/root/home   9.23G   211G     9.23G  /zroot/root/home
zroot/root/nixos  10.7G   211G     10.7G  /zroot/root/nixos

Finally, even though the boot goes to emergency mode due to the failing
mounting, simply continuing will then successfully mount the /home mount
without any further issues in use. So this is a somewhat minor issue.

Steps To Reproduce

Steps to reproduce the behavior:

1. Use the above fileSystems config with correctly partitioned single-drive setup
2. Install nixos and reboot
3. Boot fails with the above error log

However, the issue seems to appear somewhat sporadically during reboots.
Currently, it occurs every time I boot the machine. But when I tried to debug
the issue, sometimes the issue occurred and sometimes not even with the same
config.

Expected behavior

I expected the boot to succeed.

Notify maintainers

zfs package maintainers (hopefully correct):

@jcumming @jonringer @wizeman @globin

Additional info

I first posted this issue in nix-community/disko
(nix-community/disko#114) as I used it for
partitioning and mounting the drives. But the relevant fileSystems
config should be the one pasted above (printed from the final config
produced by disko). For additional info you can check the issue thread
there (e.g. some settings I have tried to use to debug the issue with no
success).

As mentioned in that thread, I suspect there is a race condition with
some systemd service that creates the empty user directory.

My host configuration with relevant parts:

• default.nix
  i.e. configuration.nix
• hardware-configuration.nix
• disko
  partitions.nix

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

➜ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 5.15.86, NixOS, 23.05 (Stoat), 23.05.20230113.befc839`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.12.0`
- nixpkgs: `/nix/store/ci0324339l5i3x7q68vigpfcf1jicsgs-source`

[wizeman]

This might be unrelated, but the first thing I would try would be to do the following:

$ zfs set mountpoint=legacy zroot/root/home

Otherwise, it might be possible that /home is trying to be mounted by both systemd and zfs during boot.

I usually recommend disabling zfs's auto-mount functionality (by setting mountpoint=legacy on all ZFS filesystems that you want to mount during boot) and using systemd exclusively to auto-mount filesystems during boot (using NixOS's fileSystems config option). The reason is that systemd can correctly mount filesystems in their proper order when you have a mix of ZFS and non-ZFS filesystems.

If that alone doesn't fix it, I would try the following:

1. Login as root
2. Log out all other users
3. Run umount /home a few times (until umount complains "not mounted")
4. Make sure that /home is not mounted by running grep home /proc/mounts and verifying that the output of that command is empty
5. Check to see if there are files/directories in /home, including hidden files (e.g. try ls -la /home, for instance)

If there are any files or directories in /home at this point, these files/directories are in your root filesystem rather than your home filesystem, so you should move them out of the way, mount /home and then move them back into their place.

[wizeman]

Also, I would try removing all the zfsutil mount options. From the description, it doesn't sound like this should be set by users (e.g. when mounting ZFS filesystems via mount or via systemd).

It sounds like zfsutil is an internal ZFS option, set by the zfs command automatically when ZFS mounts a filesystem.

[nialov]

This might be unrelated, but the first thing I would try would be to do the following:

$ zfs set mountpoint=legacy zroot/root/home

Otherwise, it might be possible that /home is trying to be mounted by both systemd and zfs during boot.

I usually recommend disabling zfs's auto-mount functionality (by setting mountpoint=legacy on all ZFS filesystems that you want to mount during boot) and using systemd exclusively to auto-mount filesystems during boot (using NixOS's fileSystems config option). The reason is that systemd can correctly mount filesystems in their proper order when you have a mix of ZFS and non-ZFS filesystems.

I am unable to test if this would work currently. Will get back to it later hopefully.

However, I am also interested if these are issues nix-community/disko should take into account when giving examples or guidance on creating a partition layout? (@Mic92 @Lassulus ). Understandably, partitioning can be complex enough that not all cases or invalid layouts can be explored, so sorry for the tag if this is out-of-scope of disko.

If that alone doesn't fix it, I would try the following:

1. Login as root

2. Log out all other users

3. Run `umount /home` a few times (until umount complains "not mounted")

4. Make sure that /home is not mounted by running `grep home /proc/mounts` and verifying that the output of that command is empty

5. Check to see if there are files/directories in /home, including hidden files (e.g. try `ls -la /home`, for instance)

If there are any files or directories in /home at this point, these files/directories are in your root filesystem rather than your home filesystem, so you should move them out of the way, mount /home and then move them back into their place.

I did this during my debugging, the issue is an empty user directory in /home. E.g. /home/nialov. No other files seem to exist that would conflict with the /home mount. Deleting that directory makes the mount succeed as expected. But the directory is created again during a new reboot. If the systemd service that creates that directory could be known, I suppose it could be modified by the user to not create that directory or wait for the zfs-mount service etc.

[wizeman]

I did this during my debugging, the issue is an empty user directory in /home. E.g. /home/nialov.

But the directory is created again during a new reboot.

I think this issue that you mentioned is known: #6481

However, judging from the following systemd message from your log, this should not be preventing /home from being mounted by systemd:

Jan 14 09:37:46 nixos-desktop systemd[1]: home.mount: Directory /home to mount over is not empty, mounting anyway.

The failure seems to come from the zfs-mount.service unit, which indicates that it's ZFS's auto-mount functionality which is failing.

I think changing the mountpoint property to legacy would work around this problem, by preventing ZFS from racing with systemd in mounting ZFS filesystems during boot.

Alternatively (i.e. if you prefer ZFS to mount the filesystem instead of systemd), you could try removing /home from fileSystems in the NixOS configuration instead of setting mountpoint to legacy.

However, generally speaking I recommend using systemd to mount ZFS filesystems due to systemd being aware of non-ZFS filesystems and thus being able to mount all filesystems in their proper order.

[nialov]

Thank you, I opted to not using a dataset for the /home/ to still continue using zfs-mount automount functionality. Switching to legacy seems to be the only way to avoid the race condition.

Can the automount functionality and using legacy mountpoints be mixed freely? I.e. using legacy for /home.

This issue can be considered "fixed" for me as I now know what to avoid what the expected errors might be. You can close it if deemed suitable or keep it open for future reference. I suppose this is a quite a major caveat when using the automount functionality of zfs.

[wizeman]

Thank you, I opted to not using a dataset for the /home/ to still continue using zfs-mount automount functionality. Switching to legacy seems to be the only way to avoid the race condition.

I think you should be able to use ZFS auto-mount for /home as long as you don't add it to the fileSystems option in your configuration.nix. As far as I understand, it's the latter that makes systemd try to mount the filesystem. So you should use either one or the other: either use ZFS auto-mount, or use systemd to mount it (with the fileSystems option).

I think the race condition should only arise when both ZFS and systemd try to mount the same filesystem(s) or when a ZFS filesystem is mounted on top of a systemd-mounted one (or vice-versa), except when mounting on top of the root filesystem which should not be an issue (I think).

Can the automount functionality and using legacy mountpoints be mixed freely? I.e. using legacy for /home.

As far as I know, yes.

[nialov]

Thank you for the clarification (and patience to explain these things! :D) as I had misunderstood some parts!

But if I do not add /home as a fileSystems option, there would be no reference to it in the NixOS configuration, correct?

[wizeman]

But if I do not add /home as a fileSystems option, there would be no reference to it in the NixOS configuration, correct?

Yes, but as far as I know, that only means that NixOS will not configure systemd to mount/unmount this filesystem automatically. However, ZFS should do it instead.

From the point of view of NixOS, it will look like /home was mounted manually, I think.

[ghost]

I seem to be unable to reproduce this issue with Root on ZFS guide. That guide uses a separate dataset for /home and mounts it with legacy mount point in configuration.nix.

[nialov]

I seem to be unable to reproduce this issue with Root on ZFS guide. That guide uses a separate dataset for /home and mounts it with legacy mount point in configuration.nix.

As I understand it, this issue is not a problem if you use /home (or other similar directory systemd uses) as a legacy mount point. In good guides and documentation this should be mentioned to avoid this issue as was done in the guide you linked.

[Shados]

The non-empty mount point bit is a red herring, I see the same issue (mountpoint or dataset is busy leading to local-fs.target failure) with one specific dataset as well, but I am fairly certain it is some sort of odd race condition with systemd mounting.

Also, unfortunately nixos/modules/system/boot/stage-1.nix makes use of the zfsutil option to determine whether or not the mount.zfs helper is required to mount a given filesystem. It probably shouldn't; we should likely just always use the helper for zfs filesystems, relying on fsType = zfs instead of a mount point.

I think the best solution to this is likely implementing #62644, as that would allow sane integration of systemd mounts with zfs mount configuration, without having to fall back to using only legacy mounts. I'm testing it out locally at the moment, and so far so good, but given the issue is racey to begin with I can't be certain as of yet that it is solved...