ZFS bind mounts on fstab makes mountall try to bind mount before ZFS FSes

[smammy]

I am using the stable ZFS PPA. On my system, fstab entries that bind mount ZFS filesystems to directories on my (non-ZFS) root filesystem cause mountall to fail to mount any ZFS filesystems.

# zpool status
  pool: storage
 state: ONLINE
  scan: scrub repaired 0 in 10h57m with 0 errors on Tue Mar  8 13:58:59 2016
config:

        NAME                                           STATE     READ WRITE CKSUM
        storage                                        ONLINE       0     0     0
          mirror-0                                     ONLINE       0     0     0
            ata-WDC_WD2002FAEX-007BA0_WD-WCAY01244342  ONLINE       0     0     0
            ata-TOSHIBA_DT01ACA200_23PJ6L1GS           ONLINE       0     0     0
          mirror-1                                     ONLINE       0     0     0
            ata-TOSHIBA_DT01ACA200_43G0BWXAS           ONLINE       0     0     0
            ata-WDC_WD20EZRX-00D8PB0_WD-WCC4NCLPADZT   ONLINE       0     0     0
        spares
          ata-TOSHIBA_DT01ACA200_43G0AYYAS             AVAIL

errors: No known data errors

# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
storage                  2.65T   887G    21K  /storage
storage/backuppc          266G   887G   266G  /storage/backuppc
storage/data             1.91T   887G  1.91T  /storage/data
storage/plexmediaserver  23.2G  16.8G  23.2G  /storage/plexmediaserver
storage/timecapsule       464G   286G   464G  /storage/timecapsule
storage/vault-home        228K   887G   228K  /storage/vault-home

# cat /etc/fstab
proc                                      /proc           proc    nodev,noexec,nosuid                      0       0
UUID=b2c87690-adc4-4b11-be8b-60fa50fecfd9 /               ext4    relatime,user_xattr,errors=remount-ro    0       1

# Bind mounts for NFSv4
/srv/pi/root             /srv/nfs4/pi              none bind 0 0
/storage/data            /srv/nfs4/data            none bind 0 0

When I boot, /storage doesn't get mounted, and mountall complains about not being able to mount /srv/nfs4/data. Here is the output of mountall --debug (with Plymouth connection warnings removed):

parse_filesystems: reading filesystems
parse_filesystems: sysfs (nodev)
parse_filesystems: rootfs (nodev)
parse_filesystems: ramfs (nodev)
parse_filesystems: bdev (nodev)
parse_filesystems: proc (nodev)
parse_filesystems: cgroup (nodev)
parse_filesystems: cpuset (nodev)
parse_filesystems: tmpfs (nodev)
parse_filesystems: devtmpfs (nodev)
parse_filesystems: debugfs (nodev)
parse_filesystems: securityfs (nodev)
parse_filesystems: sockfs (nodev)
parse_filesystems: pipefs (nodev)
parse_filesystems: anon_inodefs (nodev)
parse_filesystems: devpts (nodev)
parse_filesystems: ext3
parse_filesystems: ext2
parse_filesystems: ext4
parse_filesystems: hugetlbfs (nodev)
parse_filesystems: vfat
parse_filesystems: ecryptfs (nodev)
parse_filesystems: fuseblk
parse_filesystems: fuse (nodev)
parse_filesystems: fusectl (nodev)
parse_filesystems: pstore (nodev)
parse_filesystems: mqueue (nodev)
parse_filesystems: zfs (nodev)
parse_filesystems: rpc_pipefs (nodev)
parse_filesystems: nfs (nodev)
parse_filesystems: nfs4 (nodev)
parse_filesystems: nfsd (nodev)
parse_fstab: updating from /lib/init/fstab
update_mount: /: / /dev/root rootfs defaults check
update_mount: /proc: /proc none proc nodev,noexec,nosuid
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc none binfmt_misc nodev,noexec,nosuid,optional
update_mount: /sys: /sys none sysfs nodev,noexec,nosuid
update_mount: /sys/fs/cgroup: /sys/fs/cgroup none tmpfs optional,uid=0,gid=0,mode=0755,size=1024
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /sys/firmware/efi/efivars: /sys/firmware/efi/efivars none efivarfs optional
update_mount: /spu: /spu none spufs gid=spu,optional
update_mount: /dev: /dev none devtmpfs,tmpfs mode=0755
update_mount: /dev/pts: /dev/pts none devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /tmp: /tmp none none defaults
update_mount: /run: /run none tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /run/user: /run/user none tmpfs nodev,noexec,nosuid,size=104857600,mode=0755
update_mount: /sys/fs/pstore: /sys/fs/pstore none pstore optional
parse_fstab: updating from /etc/fstab
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /: / UUID=b2c87690-adc4-4b11-be8b-60fa50fecfd9 ext4 relatime,user_xattr,errors=remount-ro check
update_mount: /srv/nfs4/pi: /srv/nfs4/pi /srv/pi/root none bind
update_mount: /srv/nfs4/data: /srv/nfs4/data /storage/data none bind
parse_zfs_list: parsing ZFS list
update_mount: storage: /storage storage zfs zfsutil
update_mount: storage/backuppc: /storage/backuppc storage/backuppc zfs zfsutil
update_mount: storage/data: /storage/data storage/data zfs zfsutil
update_mount: storage/plexmediaserver: /storage/plexmediaserver storage/plexmediaserver zfs zfsutil
update_mount: storage/timecapsule: /storage/timecapsule storage/timecapsule zfs zfsutil
update_mount: storage/vault-home: /storage/vault-home storage/vault-home zfs zfsutil
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/mapper/vg0-keep ext4 relatime,user_xattr,errors=remount-ro check
update_mount: /sys/fs/cgroup: /sys/fs/cgroup none tmpfs optional,uid=0,gid=0,mode=0755,size=1024
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /run/user: /run/user none tmpfs nodev,noexec,nosuid,size=104857600,mode=0755
update_mount: /sys/fs/pstore: /sys/fs/pstore none pstore optional
update_mount: /srv/nfs4/pi: /srv/nfs4/pi /dev/mapper/vg0-keep ext4 bind
update_mount: /run/rpc_pipefs: /run/rpc_pipefs rpc_pipefs rpc_pipefs rw,relatime,rw
update_mount: /sys/fs/cgroup/systemd: /sys/fs/cgroup/systemd systemd cgroup rw,nosuid,nodev,noexec,relatime,rw,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd
update_mount: /proc/fs/nfsd: /proc/fs/nfsd nfsd nfsd rw,relatime,rw
mount_policy: /proc/sys/fs/binfmt_misc: dropping unknown filesystem
mount_policy: /sys/firmware/efi/efivars: dropping unknown filesystem
mount_policy: /spu: dropping unknown filesystem
mount_policy: /proc/fs/nfsd parent is /proc
mount_policy: /sys/fs/cgroup/systemd parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/systemd prior fstab entry /run/rpc_pipefs
mount_policy: /run/rpc_pipefs parent is /run
mount_policy: storage/vault-home parent is storage
mount_policy: storage/vault-home prior fstab entry storage/timecapsule
mount_policy: storage/timecapsule parent is storage
mount_policy: storage/timecapsule prior fstab entry storage/plexmediaserver
mount_policy: storage/plexmediaserver parent is storage
mount_policy: storage/plexmediaserver prior fstab entry storage/data
mount_policy: storage/data parent is storage
mount_policy: storage/data prior fstab entry storage/backuppc
mount_policy: storage/backuppc parent is storage
mount_policy: storage/backuppc prior fstab entry storage
mount_policy: storage can be mounted while root readonly
mount_policy: storage prior fstab entry /srv/nfs4/data
mount_policy: /srv/nfs4/data parent is /
mount_policy: /srv/nfs4/data parent is / (mount /storage/data)
mount_policy: /srv/nfs4/pi parent is /
mount_policy: /sys/fs/pstore parent is /sys
mount_policy: /run/user parent is /run
mount_policy: /run/shm parent is /run
mount_policy: /run/lock parent is /run
mount_policy: /run can be mounted while root readonly
mount_policy: /tmp parent is /
mount_policy: /dev/pts parent is /dev
mount_policy: /dev can be mounted while root readonly
mount_policy: /sys/kernel/security parent is /sys
mount_policy: /sys/kernel/debug parent is /sys
mount_policy: /sys/fs/fuse/connections parent is /sys
mount_policy: /sys/fs/cgroup parent is /sys
mount_policy: /sys can be mounted while root readonly
mount_policy: /proc can be mounted while root readonly
tag_mount: / is root filesystem
/ is local
/proc is virtual
/sys is virtual
/sys/fs/cgroup is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/run/user is virtual
/sys/fs/pstore is virtual
/srv/nfs4/pi is local
/srv/nfs4/data is local
storage is local
storage/backuppc is local
storage/data is local
storage/plexmediaserver is local
storage/timecapsule is local
storage/vault-home is local
/run/rpc_pipefs is virtual
/sys/fs/cgroup/systemd is virtual
/proc/fs/nfsd is virtual
mounted: /
fsck_update: updating check priorities
mounted: /proc
fsck_update: updating check priorities
mounted: /sys
fsck_update: updating check priorities
mounted: /sys/fs/cgroup
fsck_update: updating check priorities
mounted: /sys/fs/fuse/connections
fsck_update: updating check priorities
mounted: /sys/kernel/debug
fsck_update: updating check priorities
mounted: /sys/kernel/security
fsck_update: updating check priorities
mounted: /dev
fsck_update: updating check priorities
mounted: /dev/pts
fsck_update: updating check priorities
mounted: /run
fsck_update: updating check priorities
mounted: /run/lock
fsck_update: updating check priorities
mounted: /run/shm
fsck_update: updating check priorities
mounted: /run/user
fsck_update: updating check priorities
mounted: /sys/fs/pstore
fsck_update: updating check priorities
mounted: /srv/nfs4/pi
fsck_update: updating check priorities
mounted: /run/rpc_pipefs
fsck_update: updating check priorities
mounted: /sys/fs/cgroup/systemd
fsck_update: updating check priorities
mounted: /proc/fs/nfsd
fsck_update: updating check priorities
control_server_open: Mountall0_1.Server started at address: unix:abstract=/com/ubuntu/mountall/server/
run_fsck: /tmp: no check required
mounting event sent for /tmp
run_fsck: /srv/nfs4/data: no check required
mounting event sent for /srv/nfs4/data
try_mount: storage waiting for /srv/nfs4/data
try_mount: storage/backuppc waiting for /storage
try_mount: storage/data waiting for /storage
try_mount: storage/plexmediaserver waiting for /storage
try_mount: storage/timecapsule waiting for /storage
try_mount: storage/vault-home waiting for /storage
udev_catchup: catching up
try_udev_device: block /dev/sdg1 cb1c6561-6029-6866-ba21-a7bdd98c8760 keep:0
try_udev_device: block /dev/sdf1 cb1c6561-6029-6866-ba21-a7bdd98c8760 keep:0
try_udev_device: block /dev/sda 7483917943835040783 storage
try_udev_device: block /dev/sdb 7483917943835040783 storage
try_udev_device: block /dev/sdc 7483917943835040783 storage
try_udev_device: block /dev/sde 7483917943835040783 storage
try_udev_device: block /dev/dm-0 b2c87690-adc4-4b11-be8b-60fa50fecfd9 Keep
try_udev_device: / by link /dev/mapper/vg0-keep
run_fsck: /: mounted filesystem
try_udev_device: /srv/nfs4/pi by link /dev/mapper/vg0-keep
run_fsck: /srv/nfs4/pi: no check required
try_udev_device: block /dev/dm-2 a73a23dc-6bbe-4816-9c86-cffa744be694 Media
try_udev_device: block /dev/dm-3 3b60fea7-d6b1-40fa-8ff8-e8abdb195c7c Backup
try_udev_device: block /dev/dm-4 09607200-d4e6-4e7b-aaa2-c3a60ca7b20d Deepnet
try_udev_device: block /dev/md0 lAjPcz-Fe2J-agke-9nCA-RVbK-AdEI-JgACal (null)
mounted event handled for /
swap finished
local 1/10 remote 0/0 virtual 0/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys
local 1/10 remote 0/0 virtual 1/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/fs/cgroup
local 1/10 remote 0/0 virtual 2/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/fs/fuse/connections
local 1/10 remote 0/0 virtual 3/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/kernel/security
local 1/10 remote 0/0 virtual 4/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /proc
local 1/10 remote 0/0 virtual 5/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/kernel/debug
local 1/10 remote 0/0 virtual 6/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /dev/pts
local 1/10 remote 0/0 virtual 7/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /run/lock
local 1/10 remote 0/0 virtual 8/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /run/shm
local 1/10 remote 0/0 virtual 9/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /run/user
local 1/10 remote 0/0 virtual 10/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/fs/pstore
local 1/10 remote 0/0 virtual 11/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /run/rpc_pipefs
local 1/10 remote 0/0 virtual 12/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /sys/fs/cgroup/systemd
local 1/10 remote 0/0 virtual 13/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /proc/fs/nfsd
local 1/10 remote 0/0 virtual 14/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /dev
local 1/10 remote 0/0 virtual 15/16 swap 0/0
fsck_update: updating check priorities
try_mount: storage waiting for /srv/nfs4/data
try_mount: storage/backuppc waiting for /storage
try_mount: storage/data waiting for /storage
try_mount: storage/plexmediaserver waiting for /storage
try_mount: storage/timecapsule waiting for /storage
try_mount: storage/vault-home waiting for /storage
mounting event handled for /tmp
run_mount: /tmp: placeholder
mounted: /tmp
fsck_update: updating check priorities
mounting event handled for /srv/nfs4/data
mounting /srv/nfs4/data
spawn: mount -n -t none -o bind /storage/data /srv/nfs4/data
mount: special device /storage/data does not exist
mountall: mount /srv/nfs4/data [8308] terminated with status 32
mountall: Filesystem could not be mounted: /srv/nfs4/data
mountall: Skipping mounting /srv/nfs4/data since Plymouth is not available
spawn: mount /srv/nfs4/data [8308]
skip_mount: /srv/nfs4/data
skip_mount: storage
skip_mount: storage/backuppc
skip_mount: storage/data
skip_mount: storage/plexmediaserver
skip_mount: storage/timecapsule
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
skip_mount: storage/backuppc
skip_mount: storage/data
skip_mount: storage/plexmediaserver
skip_mount: storage/timecapsule
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
skip_mount: storage/data
skip_mount: storage/plexmediaserver
skip_mount: storage/timecapsule
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
skip_mount: storage/plexmediaserver
skip_mount: storage/timecapsule
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
skip_mount: storage/timecapsule
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
skip_mount: storage/vault-home
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
local 1/3 remote 0/0 virtual 15/16 swap 0/0
mounted event handled for /srv/nfs4/pi
local 2/3 remote 0/0 virtual 15/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /run
virtual finished
remote finished
local 2/3 remote 0/0 virtual 16/16 swap 0/0
fsck_update: updating check priorities
mounted event handled for /tmp
local finished
All filesystems mounted
local 3/3 remote 0/0 virtual 16/16 swap 0/0
fsck_update: updating check priorities

try_mount has somehow gotten the idea that storage (the ZFS base filesystem) depends on /srv/nfs4/data, which in reality a bind mount target:

try_mount: storage waiting for /srv/nfs4/data

And then a little later on, it tries to mount /srv/nfs4/data, but of course can't, because /storage/data doesn't exist yet:

mounting event handled for /srv/nfs4/data
mounting /srv/nfs4/data
spawn: mount -n -t none -o bind /storage/data /srv/nfs4/data
mount: special device /storage/data does not exist
mountall: mount /srv/nfs4/data [8308] terminated with status 32
mountall: Filesystem could not be mounted: /srv/nfs4/data
mountall: Skipping mounting /srv/nfs4/data since Plymouth is not available
spawn: mount /srv/nfs4/data [8308]
skip_mount: /srv/nfs4/data
skip_mount: storage
skip_mount: storage/backuppc
skip_mount: storage/data
skip_mount: storage/plexmediaserver
skip_mount: storage/timecapsule
skip_mount: storage/vault-home

If I remove the bind mount from /etc/fstab, everything works fine (except my NFSv4 exports, of course).

[dajhorn]

@smammy,

This is a known corner case for mountall, and one of the reasons why it was deprecated in Ubuntu for the systemd stack.

Two potential solutions are:

• Reverse the mount and bind points. (ie: Set mountpoint=/srv/nfs4/data on the dataset and bind it back to /storage/data in the /fstab.)
• Do the binding in an upstart job that listens for the MOUNTED=/storage/data event and depends on the NFS service.

Past that, if you're building a new system and need this kind of configuration, then think about using Xenial.