Troubleshooting mountall

On Ubuntu systems, and its derivatives like Linux Mint, the /sbin/mountall utility mounts filesystems at system start. The PPA provides a mountall package that is enhanced to recognize ZFS datasets.

On regular Debian systems, mountall is not required and should not be installed. Do not install mountall if the system does not already have mountall installed. Do not force a mountall installation.

Step 1: Turn off ZFS_MOUNT.

• Ensure that ZFS_MOUNT='no' is set in the /etc/default/zfs file.

Step 2: Check that the PPA packages are installed.

$ apt-cache policy mountall ubuntu-zfs
mountall:
  Installed: 2.36.2-zfs1
  Candidate: 2.36.2-zfs1
  Version table:
 *** 2.36.2-zfs1 0
       1001 http://ppa.launchpad.net/zfs-native/daily/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
     2.36 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
ubuntu-zfs:
  Installed: 7~precise
  Candidate: 7~precise
  Version table:
 *** 7~precise 0
       1001 http://ppa.launchpad.net/zfs-native/daily/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

The mountall package must have "zfs" in its version string. The ubuntu-zfs package is required to pin the mountall package.

Step 3: Double-check that the PPA packages are installed.

$ strings /sbin/mountall | grep ZFS
%s: parsing ZFS list

If "parsing ZFS list" does not appear in this output, then the /sbin/mountall utility will not recognize ZFS datasets.

Step 4: Enable mountall debugging messages.

• Edit the /etc/init/mountall.conf file.
• And change the exec line to this:

	exec mountall --debug --daemon $force_fsck $fsck_fix

Step 5: Report the boot.log file.

• Reboot the computer.
• And submit the /var/log/boot.log file by posting it to http://gist.github.com/ or attaching it unmodified to an email.
• The top of this file should look like this:

	Connected to Plymouth
	parse_filesystems: reading filesystems
	parse_filesystems: sysfs (nodev)
	parse_filesystems: rootfs (nodev)
	parse_filesystems: bdev (nodev)
	parse_filesystems: proc (nodev)
	parse_filesystems: cgroup (nodev)
	parse_filesystems: cpuset (nodev)
	parse_filesystems: tmpfs (nodev)
	parse_filesystems: devtmpfs (nodev)
	parse_filesystems: debugfs (nodev)
	parse_filesystems: securityfs (nodev)
	parse_filesystems: sockfs (nodev)
	parse_filesystems: pipefs (nodev)
	parse_filesystems: anon_inodefs (nodev)
	parse_filesystems: devpts (nodev)
	parse_filesystems: ext3
	parse_filesystems: ext4
	parse_filesystems: ramfs (nodev)
	parse_filesystems: hugetlbfs (nodev)
	parse_filesystems: ecryptfs (nodev)
	parse_filesystems: fuseblk
	parse_filesystems: fuse (nodev)
	parse_filesystems: fusectl (nodev)
	parse_filesystems: pstore (nodev)
	parse_filesystems: mqueue (nodev)
	parse_fstab: updating from /lib/init/fstab
	update_mount: /: / /dev/root rootfs defaults check
	update_mount: /proc: /proc none proc nodev,noexec,nosuid
	update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc none binfmt_misc nodev,noexec,nosuid,optional
	update_mount: /sys: /sys none sysfs nodev,noexec,nosuid
	update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
	update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
	update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
	update_mount: /sys/firmware/efi/efivars: /sys/firmware/efi/efivars none efivarfs optional
	update_mount: /spu: /spu none spufs gid=spu,optional
	update_mount: /dev: /dev none devtmpfs,tmpfs mode=0755
	update_mount: /dev/pts: /dev/pts none devpts noexec,nosuid,gid=tty,mode=0620
	update_mount: /tmp: /tmp none none defaults
	update_mount: /run: /run none tmpfs noexec,nosuid,size=10%,mode=0755
	update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
	update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
	parse_fstab: updating from /etc/fstab
	update_mount: /proc: /proc proc proc nodev,noexec,nosuid
	update_mount: /: / UUID=6892cb6c-d00f-48fb-bf04-833aa9ee078c ext4 errors=remount-ro check
	update_mount: UUID=a5566bef-a07d-4720-a045-ec2d935cfbcc: none UUID=a5566bef-a07d-4720-a045-ec2d935cfbcc swap sw
	update_mount: /media/floppy0: /media/floppy0 /dev/fd0 auto rw,user,noauto,exec,utf8
	parse_zfs_list: parsing ZFS list
	update_mount: /tank: /tank tank zfs zfsutil,nobootwait

The /sbin/mountall utility does not have ZFS support if the parse_zfs_list: parsing ZFS list line is missing.

Step 6: Report the dmesg file.

• Submit the /var/log/dmesg file by posting it to http://gist.github.com/ or attaching it unmodified to an email.

• The disk drives must be online before the ZFS driver is loaded. This means that lines for each pool member (sda, sdb, sdc, ...) must appear in the dmesg before the ZFS line like this:

[   12.161920] sd 2:0:0:0: [sda] Attached SCSI disk
[   12.162020] sd 2:0:1:0: [sdb] Attached SCSI disk
[   12.162111] sd 2:0:2:0: [sdc] Attached SCSI disk
[   12.162286] sd 2:0:3:0: [sdd] Attached SCSI disk
[   12.162472] sd 2:0:4:0: [sde] Attached SCSI disk
[   22.740575] SPL: Loaded module v0.6.0.91-rc13
[   22.741517] zunicode: module license 'CDDL' taints kernel.
[   22.741520] Disabling lock debugging due to kernel taint
[   23.111192]  zd0: unknown partition table
[   23.113762] ZFS: Loaded module v0.6.0.91-rc13, ZFS pool version 28, ZFS filesystem version 5
[   23.115465] SPL: using hostid 0x007f0101

• If any of the Attached lines come after the ZFS: Loaded module line, then the pool will fail to import.
• This often happens because "zfs" is in the /etc/initramfs-tools/modules file, which is incorrect for Ubuntu systems. The ZFS driver should be loaded as late as possible during system start.

Hardware Notes

Areca

Systems that require the arcsas blob driver should add it to the /etc/initramfs-tools/modules file and run update-initramfs -c -k all.

Upgrade or downgrade the Areca driver if something like RIP: 0010:[<ffffffff8101b316>] [<ffffffff8101b316>] native_read_tsc+0x6/0x20 appears anywhere in kernel log. ZoL is unstable on systems that emit this error message.

MPT2SAS

Some hardware that uses the mpt2sas driver does slow asynchronous drive initialization, such that the disks on these controllers are not visible to the Linux kernel until long after the regular system is started. The IBM M1015 and some similar OEM-branded cards that have been flashed to the reference LSI firmware are notably problematic.

Kludge this problem by adding a sleep 60 statement in the /etc/init/mountall.conf file like this:

sleep 60
exec mountall --debug --daemon $force_fsck $fsck_fix

And increase the number if the problem persists. Some controllers need two or three minutes to bring all disks online, which would be sleep 180.

Most LSI cards are perfectly compatible with ZoL, but there is no known fix if your card has this glitch. Please use different equipment until the mpt2sas incompatibility is diagnosed and fixed, or donate an affected part if you want solution sooner.