Add labelfix (zpool label recovery tool) to official tools.

[Sachiru]

Back in the old days, Jeff Bonwick created a tool to recover broken labels on zpools, called labelfix.

A copy is still located at https://gist.github.com/jjwhitney/baaa63144da89726e482

Can we perhaps add this to the official list of tools?

[kernelOfTruth]

@Sachiru you mean something like the following:

#4186 Allow -F to regenerate missing vdev labels

action: The pool cannot be "
 +              "imported due to a corrupt or missing label.\n\t"
 +              "It may be possible to reconstruct the label "
 +              "from the cache file.\n\tImport the pool with "
 +              "\"-F -o readonly=on -c <cache-file>\".\n"));

[behlendorf]

Actually this looks like a tool specifically designed to handle something like #4162. It rewrites the label with an txg consistent to that from the uberblock. This is something zpool import itself needs to handle.

#4186 is slightly different it rewrites all the labels given a cache file and it probably still needs some refinement. But the idea here is that in general these are things zpool import itself should be able to handle.

[behlendorf]

Can we perhaps add this to the official list of tools?

Adding this as a zhack subcommand seems like a reasonable this to do. This is a developer tool which shouldn't be needed by end users but sometimes it is handy to have a tool for this kind of thing.

[Sachiru]

The initial idea is a tool that rewrites the label given a cache file, yes, however the visualized end goal is a recovery tool that can rewrite a label based on several possible methods:

a) Recover from a cache file
b) Reconstruct cases of damaged partition tables by scanning the disk for pool_guid then following this method: https://forums.freebsd.org/threads/need-help-to-recover-data-from-damaged-zfs-pool.49782/#post-279702
c) Reconstruct by scanning labels from healthy members of the vdev and speculatively writing them to the failed members.
d) Reconstruct cases of damaged partition tables by scanning for the uberblock.
e) Additional methods of recovery as they are developed.

This is useful to shut up people who think that a destroyed pool means permanent loss of data and use that as a way to discredit ZFS by saying that if it's broken there's no hopes of fixing it. This also allows ZFS to have available data recovery tools, something which other FSes do have (and yes, although in an ideal world ZFS does not need data recovery tools, in an imperfect world it will be nice to have this for ZFS too).

[ilovezfs]

@Sachiru yes such a tool would be very nice to have, but I wouldn't oversell it, and I think it will largely leave the lack-of-fsck and lack-of-recovery-tools complaints entirely unaddressed.

The scenarios you described are of the ultimately very minor variety. 1) More serious problems such as spacemap corruptions, device not found I/O errors, metadata corruptions, etc. would still fall into the no-recovery-tools-available category that people complain about. 2) Unless the tool has a way to do its troubleshooting accurately in a 100% readonly mode, it will inevitably sometimes make problems worse and earn a bad reputation because many people won't follow proper forensic procedures, often because they'll claim they don't have sufficient space to image every disk. Blaming the victim in such cases will not be received well.

[richardelling]

a) Recover from a cache file

If the pool is writable, then this already exists: import the pool with cachefile set or -c option.

b) Reconstruct cases of damaged partition tables by scanning the disk for pool_guid then following this method: https://forums.freebsd.org/threads/need-help-to-recover-data-from-damaged-zfs-pool.49782/#post-279702

This isn't difficult, because the labels and uberblocks have a well-known structure.
But it really isn't a ZFS command, per se. I can see it as an option to zdb: report discovered
labels.

c) Reconstruct by scanning labels from healthy members of the vdev and speculatively writing them to the failed members.

This seems scary... what if the vdevs were repurposed?

d) Reconstruct cases of damaged partition tables by scanning for the uberblock.

Same as b above?

e) Additional methods of recovery as they are developed.

Catch all, not really actionable, though.

[Sachiru]

@ilovezfs : True, but even minor recovery tools are better than the lack of them.

@richardelling : The reconstruction via speculative writing can probably be hidden behind a (documented) command line flag, something like

zhack labelfix --reconstruct-from-vdev

And even then, inquiring from the user if they are sure or not before the command is issued. Of course, those who would recommend this method of data recovery would carefully consider if this is the appropriate solution or not, and any issues about when and where to use this will be detailed in the man page.

The catchall is only there if we find other methods, and yes, scanning for either the uberblock or the pool_guid string should both be folded into b.

[JuliaVixen]

A scenario in which this is a very useful feature to have, is a situation when you're "zpool replaced" some disks in a pool, and then something happens to the live pool, and you still have the slightly older data, on the original (older) drives you replaced... except... "zpool import" can't see the old replaced drives. Apparently the drives are "destroyed" after resilvering has completed on the new drive.

So, Let's say you have a pool like:

  pool: example
 state: ONLINE
  scan: scrub canceled on Thu Apr 28 01:14:19 2016
config:

    NAME                                             STATE     READ WRITE CKSUM
    example                                          ONLINE       0     0     0
      raidz1-0                                       ONLINE       0     0     0
        sda                                          ONLINE       0     0     0
        sdb                                          ONLINE       0     0     0
        sdc                                          ONLINE       0     0     0

And then you "zpool replace example /dev/sda /dev/sdx"
And after that's done, you "zpool replace example /dev/sdb /dev/sdy"
And finally, you "zpool replace example /dev/sdc /dev/sdz"

    NAME                                             STATE     READ WRITE CKSUM
    example                                          ONLINE       0     0     0
      raidz1-0                                       ONLINE       0     0     0
        sdx                                          ONLINE       0     0     0
        sdy                                          ONLINE       0     0     0
        sdz                                          ONLINE       0     0     0

And everything's hunky-dorey, and life is good, and then disaster strikes, and /dev/sdx and /dev/sdy are hit by a meteorite or something.

In theory, all of the data in the pool is still on the old /dev/sda, /dev/sdb, and /dev/sdc devices which were physically removed and kept in an underground bunker...

So... You should be able to plug the old /dev/sda, /dev/sdb, and /dev/sdc devices back in, and re-import the pool, or at least be able to look at the data with the "zdb" tool...

NOPE! That doesn't work, neither "zpool import", nor "zdb -e" will see the devices as belonging to any pool. However, "zdb -l /dev/sda", "zdb -lu /dev/sdb", etc. work perfectly fine. You can totally see all the labels, and the vdev_tree, and that it's a zraid, with three drives, and the GUID's and TXG's and everything.

I tried using https://gist.github.com/jjwhitney/baaa63144da89726e482 and it seemed to work, in so much as "zpool import" could see the drives. (I didn't quite actually get the drives imported yet though.)

It appears that @jjwhitney has already ported Bonwick's labelfix to the current SPL. It's all still a bit of a hack, and it would be nice if it didn't have to write back to the original device, and just do some kind of indirection to a new copy of the label elsewhere (or just do the import with the found label in the place it was found), but this would probably all be part of some forensics analysis features for zdb.

[timkgh]

+1 for having such a tool. I just detached a drive from a mirror and found out the hard way the drive is now pretty much useless, cannot import the pool on another host even though all the data is still on the drive. It is my fault for not learning about zpool split (or apparently offline the drive first and then detach would have also worked), but I think we should have a way to still import a detached drive.

[behlendorf]

Some progress has been made on this front, the zhack label sub-command can now be used to repair a damaged vdev label. It's a start, but more work is needed in order to support more targeted label modifications (such as marking a destroyed vdev label as valid again). Commit d04b5c9.

      zhack label repair device
             Repair corrupted labels by rewriting the checksum using the pre‐
             sumed valid contents of the label.

[burny0202]

Didn't appear to work for me on an accidentally detached special device.

I got error

zhack label repair /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_1TB_S6Z1NJ0W333776L

Error:
Expected the nvlist checksum magic number to not be zero

[buzzingwires]

@burny0202

A few things:

Firstly, in order to restore a detached label, you will need to invoke label repair with -u. -c can also be added to fix checksums (which is the default old behavior). But I think the software will still fail to proceed with your current issue.

zhack label repair -cu /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_1TB_S6Z1NJ0W333776L

If possible, you might consider running this on an image of the device so that you can go back to a fresh copy.

Secondly, could you run zdb -l /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_1TB_S6Z1NJ0W333776L and post the output? Feel free to redact the name and hostname fields if they are sensitive. Just let me know if they're not what you'd expect them to be.

[ggzengel]

Because of expanding VDEVs I got two different labels on VDEVs (#16144).
So there should be a tool to copy labels from one slot to an other slot.