import failure, weird unavail issue #3107
Comments
|
Bueller? Not sure if this is the case but based on my readings of similar things on the issue tracker, this isn't just a "help me!" - I'd really like to know what's going on here and I think I have some legitimate bugs... so it's ultimately a combo posting:
All throughout this endeavor I've dmesg'd regularly - there has been no hardware messages past boot and nothing erroneous.. |
|
Small updates: I've been cruising the ML and issue trackers and have spotted some similiar things but really nothing figuring out what causes these problems - they're just worked around (while being a little more managable), given up on, or replacing the device / rebooting seems to work. Since then I've installed debug builds from git sources and here's some output of and here's another listing of If I should move this to the ML I'd be more than happy to but the lack of response here is striking me as ignoring... does any dev care to chime in yet? |
|
|
|
@setharnold Yes, I tried that numerous times before this got all crazy - back when I first found the dev paths changing on upgrade to 3.16. I exported the pool and imported it again with that command It didn't work at the time - only symlinking would allow the drives be found. Eventually I started replacing the drives one at a time which as I said wasn't too problematic till this instance. I suspected this might be related to why it isn't/wastn't before "corruption" picking up the devices that were available. Is there any debugging this mechanism that is probing for the drives available to see why it fails? |
|
I suspect this stems from using an older FreeBSD which did not create partitions, @ryao explained how he helped fix a related issue in #3024. That issue is not too similar to this one, though. Are there partitions on the devices marked UNAVAIL? If so, it could help to |
|
@dasjoe - yes, I started with Freebsd but the machine had to do more than be a nas so I moved it over to ZoL once it was looking usable. It was only the first vdev group that I created with zfs. I tried to keep everything looking the same with gdisk/cgdisk as I added in pool 2 a year+ ago. @ryao 's fix seems to be something worth trying as I've replaced these disks and pool 1 had some experimental setups on it before it went stable - I've always used the same poolname for them since there was just one instance at a time. I tried rming the raw device and there was a small change, the last drive of raidz2-0 isn't marked corrupted, just unavail now: I also added a gist for parted print all: https://gist.github.com/nevion/179eade4f8b42773126f @ryao - do you think I should try your approach in #3024 to deal with those corrupted data vdevs? |
|
@nevion My apologies for the delayed response, but most of us are very busy at the moment. Doing the procedure from #3024 would make things worse if it is not the right thing to do, but it is worth a try with some precautions. In specific, you should make images of the disks that are unavailable. In this situation, just the GPT partition tables need to be imaged, so the first 10MB of the front and last 10MB of the back should be more than sufficient. It is also possible to backup the GPT partition tables with That said, I would be happy to help you in IRC between 5pm to 9pm EST. If this pool belongs to a business, I would like to be paid for my effort (I charge $2000 per attempt) while if it is your own or belongs to a non-profit organization, I do not charge. |
|
@ryao left some messages to you on irc so far I've taken one drive from the pool, backed up the gpt header w/ sgdisk and dumped the list again with zdb -l it looks to me it is seeing the old freebsd vdev... a mix of it rather, I forgot linux does not have /dev/gpt and I noticed them in the zdb list |
|
I've added some additional gist files: This shows zdb -lu for each partition currently plugged into the pool: This breaks down devices into controller and status to determine hardware correlations (I did not spot any) @ryao astutely observed the txg's of some devices are higher than in others. He suspected my HBA's of having a write cache. I confirmed with jasonwc on irc that with the LSI 2008 IT firmwares, these cards don't do write-caching - so that theory is out. By the way, the cards are running IT firmware version PH14 and the card models are Supermicro AOC-USAS2-L8i. So far, @ryao has thought that one thing worth a shot would be rewinding to the transaction present to all drives (rewinding the drives into the past). We've not yet done that and are making sure it's the best strategy. |
|
I updated to git HEAD to snag this patch. zpool import still get's confused with the wwn / scsi-3xxx entries for my harddrives. As such I move them somewhere else while testing importing. I'm still getting one drive missing which might have been labelcleared, the other I have disconnected and will reconnect momentarily. and doing zpool import -d /dev/disk/by-id/ again: I'm about to go resolve the bottom missing drive which also might have been label cleared. As is the drives here all show online - should I try importing them whether or not that missing drive gets detected? @behlendorf I also want to be cautious because while debugging with @ryao - several of my drives, according to zdb -lu, have different txg ids, greater than a few hundred. I attribute this to zfs starting a resilvering - I recall it saying it didn't have enough drives at the time to import but it was resilvering (automatically) anyway. |
|
small update, plugged in last drive of pool and we currently are here: |
|
@nevion that's looking good so far. The Regards, |
|
@dasjoe It's this drive: https://gist.github.com/nevion/e3a81206a8bc40f47e60#file-scsi-sata_wdc_wd30efrx-68a_wd-wcc1t0804382-part1 Actually I don't know why I forgot this but here is a zfs history before the pool started getting bogged down with "corruption": https://gist.githubusercontent.com/nevion/0bcb7010e52184b07576/raw/zfshist.txt I'm not sure I ever succeeded in labelclearing although I typed commands to do it - I always got errors about the drive being active in the pool, event though it was marked as offline in status. I tricked it into that by not creating the old symlinks the zfs cache file prior to 3.16 was expecting for those drives I wanted to clear and replace. You can see the updates in the uber block for the drive and the log superseding it if I read it right. Still waiting on expert opinion to tell me if I should go ahead and attempt import or collect diagnostics etc. The data on the drives matters. |
|
OK well I've imported it successfully. It's resilvering as expected. So far I've gotten the following status and error list (note the metadata 0xb6 error): |
|
The resilver completed without further incidence. I then replaced the "corrupted" vdev with the same disk and that completed without issue and then I scrubbed the pool. So it's a full recovery. |
When using 'zpool import' to scan for available pools prefer vdev names which reference vdevs with more valid labels. There should be two labels at the start of the device and two labels at the end of the device. If labels are missing then the device has been damaged or is in some other way incomplete. Preferring names with fully intact labels helps weed out bad paths and improves the likelihood of being able to import the pool. This behavior only applies when scanning /dev/ for valid pools. If a cache file exists the pools described by the cache file will be used. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chris Dunlap <cdunlap@llnl.gov> Closes openzfs#3145 Closes openzfs#2844 Closes openzfs#3107
When using 'zpool import' to scan for available pools prefer vdev names which reference vdevs with more valid labels. There should be two labels at the start of the device and two labels at the end of the device. If labels are missing then the device has been damaged or is in some other way incomplete. Preferring names with fully intact labels helps weed out bad paths and improves the likelihood of being able to import the pool. This behavior only applies when scanning /dev/ for valid pools. If a cache file exists the pools described by the cache file will be used. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chris Dunlap <cdunlap@llnl.gov> Closes openzfs#3145 Closes openzfs#2844 Closes openzfs#3107
and others
So I use mpt2sas for my 2 disk arrays. Since kernel 3.16 or so, there has been some change that causes the drives to export differently under /dev/disk/by-id as such:
ata-TOSHIBA_DT01ACA300_Z2T3B8LAS-part1 -> scsi-SATA_TOSHIBA_DT01ACA3_Z2T3B8LAS-part1
Apparently the way to fix device name based issue this is to fake the death of the drives (one at a time) and replace it with another... like it's self. I've done this for a number of disks and didn't really encounter an issue until today. Keep in mind I only was trying to replace the last drive of each raidz group. Somehow this has resulted in UNAVAIL OFFLINE and now UNAVAIL corrupted data. The machine has ECC ram and I've not noted anything else out of the ordinary.
The procedure I've been using is to start from a clean boot - auto import of the raid fails because the ata devices do not exist under /dev/disk/by-id. I then create the symlinks to them based on what zfs status reports for the vdev and what matches the serial numbers and starts with scsi-sata under /dev/disk/by-id.except for the ones which I will be replacing. I then zfs replace the devices in question - usually I'm greeted with a error saying that drive is still part of the group even though it is marked unavail. So to get around that I run zfs export, then zfs labelclear vdev and import the pool again. After that was done today however somehow device 5 (10976772185152087831 and 14507462425068499796) both started reporting unavail and after a couple of reboots it said corrupted data and somehow spread to device 6 in both pools. I'll also point out somehow the vdev paths it is using change on import (when it is having trouble importing devices) to numbers (like above) and scsi- - It has done this (erroneous?) path change all throughout this process of device replacing.
I'm completely baffled at this result. As I poked around and rebooted I've noticed the paths keep changing. All the devices exist though though - I can cat/dd from them just fine.
Versions: zfs-0.6.3_git.1423692317.e02b533 , kernel 3.16.7 on OpenSUSE 13.2.
Where do I go from here?
The text was updated successfully, but these errors were encountered: