ZFS umount errors during shutdown

[Izorkin]

Hi.

I'm trying to use separate mountpoints for MySQL in nixos, like this:

/ = ztank/nixos/root1
/nix = ztank/nixos/nix
/tmp = ztank/nixos/tmp
/var/tmp = ztank/nixos/tmp_var
/home/user = ztank/home/user
/var/lib/mysql = ztank/data/mysql
/var/lib/mysql/ibdata = ztank/data/mysql/ibdata
/var/lib/mysql/iblogs = ztank/data/mysql/iblogs

As result, I observe the following errors during system shutdown:

Feb 05 12:09:39 nixos-vmware systemd[1]: ^[[0;1;39mvar-lib-mysql-iblogs.mount: Mount process exited, code=exited status=32
Feb 05 12:09:39 nixos-vmware systemd[1]: Failed unmounting /var/lib/mysql/iblogs.
Feb 05 12:09:39 nixos-vmware systemd[1]: ^[[0;1;39mrun-user-1000.mount: Mount process exited, code=exited status=32
Feb 05 12:09:39 nixos-vmware systemd[1]: Failed unmounting /run/user/1000.
Feb 05 12:09:40 nixos-vmware systemd[1]: ^[[0;1;39mvar-lib-mysql.mount: Mount process exited, code=exited status=32
Feb 05 12:09:40 nixos-vmware systemd[1]: Failed unmounting /var/lib/mysql.

Filesystem mount options in hardware-configuration.nix:

  fileSystems."/" = {
    device = "ztank/nixos/root1";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/4046e960-93b6-4f13-a5d5-70ae52932573";
    fsType = "ext3";
    options = [ "relatime" ];
  };

  fileSystems."/nix" = {
    device = "ztank/nixos/nix";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/etc/nixos" = {
    device = "ztank/nixos/cfg";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/tmp" = {
    device = "ztank/nixos/tmp";
    fsType = "zfs";
    options = [ "nosuid" "nodev" "noatime" ];
  };

  fileSystems."/var/tmp" = {
    device = "ztank/nixos/tmp_var";
    fsType = "zfs";
    options = [ "nosuid" "nodev" "noatime" "noexec" ];
  };

  fileSystems."/home/user" = {
    device = "ztank/home/user";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/root" = {
    device = "ztank/home/root";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/var/lib/mysql" = {
    device = "ztank/data/mysql";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/var/lib/mysql/ibdata" = {
    device = "ztank/data/mysql/ibdata";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

  fileSystems."/var/lib/mysql/iblogs" = {
    device = "ztank/data/mysql/iblogs";
    fsType = "zfs";
    options = [ "relatime" "xattr" "posixacl" ];
  };

I also tried the following:

zfs set mountpoint=/var/lib/mysql /var/lib/mysql
zfs set mountpoint=/var/lib/mysql/ibdata ztank/data/mysql/ibdata
zfs set mountpoint=/var/lib/mysql/iblogs ztank/data/mysql/iblogs

But it did not help. Unmounting is done by systemd, and the same errors occur.

Method, recommended in the following comment, does not help either: #21928 (comment)

How can I ensure that unmounting of filesystems is done correctly? Where/how can I configure systemd to wait for full stop of MySQL before unmounting ZFS file systems?

• system: "x86_64-linux"
• host os: Linux 4.15.0, NixOS, 18.03pre126729.2e4aded3669 (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 1.11.16
• channels(root): "nixos-18.03pre126729.2e4aded3669"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[Mic92]

Do you use legacy mountpoints as described in the wiki: https://nixos.wiki/wiki/NixOS_on_ZFS#How_to_use_it (zfs set mountpoint=legacy filesystem)?
Otherwise mountpoints cannot be correctly ordered on startup and shutdown.
This might change in future, when we have a fstab generator for zfs: openzfs/zfs#4943

[Mic92]

Usually systemd makes sure, that all processes are killed before it unmounts local filesystems.

[Mic92]

The errno is quite strange. EBUSY should be returned if mysql is still accessing the device.

[Izorkin]

Yes, me used legacy mountpoints

zfs list | grep mysql
ztank/data/mysql          582K  8,31G    25K  legacy
ztank/data/mysql/ibdata   527K  8,31G   527K  legacy
ztank/data/mysql/iblogs  29,5K  8,31G  29,5K  legacy

[Izorkin]

Is it possible to configure systemd to skip mounting/unmounting of file systems altogether?

[Mic92]

Mounting can be disabled using: fileSystems."/mnt/backup".options = ["noauto"];

[Mic92]

I also use nofail sometimes to mark a mountpoint as non-critical.

[phryneas]

Has one of your filesystems (most likely tmp) sync=disabled?
That could be caused by this bug: openzfs/zfs#7753 (comment)

[Izorkin]

Yes

zfs get sync ztank/nixos/tmp
NAME             PROPERTY  VALUE     SOURCE
ztank/nixos/tmp  sync      disabled  received

[phryneas]

Nice! I know, this issue is half a year old - but a bug fix is on the way :)

[Mic92]

We have now 0.8.0-rc3 merged into nixos, that should have this bugfix.
Can we close this issue? openzfs/zfs#7753 (comment)

[Izorkin]

I have another configuration now.
There is only such an error

фев 09 22:10:09 elven.pw umount[2381]: umount: /var: target is busy.
фев 09 22:10:09 elven.pw systemd[1]: ^[[0;1;39m^[[0;1;39mvar.mount: Mount process exited, code=exited status=32
фев 09 22:10:09 elven.pw systemd[1]: ^[[0;1;39mFailed unmounting /var.

[Izorkin]

Checked on a virtual machine

фев 11 10:31:13 nixos-local systemd[1]: Unmounting /var/lib/mysql/ibdata...
фев 11 10:31:13 nixos-local systemd[1]: Unmounting /var/lib/mysql/iblogs...
...
фев 11 10:31:13 nixos-local systemd[1]: Unmounted /var/lib/mysql/ibdata.
фев 11 10:31:13 nixos-local systemd[1]: Unmounted /var/lib/mysql/iblogs.
...
фев 11 10:31:13 nixos-local systemd[1]: Unmounting /var/lib/mysql...
...
фев 11 10:31:13 nixos-local systemd[1]: Unmounted /var/lib/mysql.
фев 11 10:31:13 nixos-local systemd[1]: Unmounting /var...
фев 11 10:31:13 nixos-local umount[25261]: umount: /var: target is busy.
фев 11 10:31:13 nixos-local systemd[1]: var.mount: Mount process exited, code=exited status=32
фев 11 10:31:13 nixos-local systemd[1]: Failed unmounting /var.
фев 11 10:31:13 nixos-local systemd[1]: Reached target Unmount All Filesystems.

/var/lib/mysql normally unmount

[phryneas]

which version of zfs & spl are you running?
modinfo zfs | grep version and modinfo spl | grep version

[Izorkin]

zfs

version:        0.8.0-rc3
srcversion:     E8ED8A9D1524780F1F823C9

spl

version:        0.8.0-rc3
srcversion:     52EA3DB565B585F98955FAE

Unmount error "/ var" probably does not apply to zfs.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[Mic92]

Is this still an issue?