PANIC: zfs: adding existent segment to range tree #15030
Labels
Type: Defect
Incorrect behavior (e.g. crash, hang)
Comments
|
Got about 12 hours from it, before it got wonky and panic again. First few entries seem to be related to The ZFS panic does not show up until a bit later: |
|
I was having similar symptoms on a TrueNAS Scale system. My backup pool of 2 disks got corrupted somehow. Scrub is clean, but any time I'm pushing a particular dataset to it, it panics. I was able to reproduce the panic by manually deleting one snapshot from the dataset. Export and reimport works, but deleting snapshots immediately panics the system and needs hard reset since normal shutdown never completes after zfs panic. Since the backup pool is somewhat important for me, I enabled |
|
I left it running for about a week with I've had no more I assume this is from zfs constantly rewriting spacemaps in condensing operation. Running the pool in recovery mode for a while eventually removed the duplicated range descriptions during the rewrite operations.... or something like that. |
|
I'm also seeing this on a pool using kernel 6.5-rc6. Here is my zdb output: https://dpaste.com/DL9KS6SQ5 I've set boot variables for |
|
@reefland Is there a way to force spacemap condensing to happen faster to try to resolve this? |
|
@satmandu - sorry, no idea. No clue how to monitor the spacemap condensing operation - trial and error. Hopefully someone more knowledgeable can answer.
|
|
Thanks. My pool has clean scrubs, but my system still panics if I mount the pool without the special zfs options... but I also haven't had the pool mounted with the special options for more than 16 hours... |
|
If rewriting the spacemaps removes the assertion, that might suggest you still have lurking excitement eventually, since in theory that should be equally upset by duplicate range insertion, no? My wild guess would be when it threw out the log after you imported the pool and unlocked the datasets with replaying disabled, it no longer had a pending insertion that was invalid. But that's just a wild guess. |
|
I hit this again... and had to set |
|
I've had this multiple times on different systems (it has not duplicated on the same system). It's just a process I follow now to recover the system. At least its a workaround. I'm no closer to finding a cause / pattern. |
|
I destroyed a zfs subvolume (that may have had this issue because that's where intense writes from docker were happening) and still have the issue when importing the parent volume. I will copy all the data to a fresh volume and recreate the volume. If I delete all the data and the volume still has these issues, could someone look at an image of the disk to help figure out how to repair this issue, even if figuring out how this damage is being created isn't proximally resolved? Would that be useful at all? |
|
@satmandu - I can't comment on your disk image question. But I've had to destroy a number of ZVOLs (extract data, delete ZVOL, recreate, restore, etc) related to this as well. I've since switched all ZVOLs to forced sync writes. Yeah, performance impact, but I haven't had any ZVOLs get corrupted since. I was frequently getting this after a ZFS panic: Multiple scrubs were not able to fix it. I'm using CRI / K3s instead of Docker, but we're probably using ZVOLs for similar reasons. |
|
I wonder if we can create a test case of amplifying read/writes to replicate the issue. Or maybe if there is an existing test case we should be trying on our hardware? This sounds like it should be replicable in a test environment where one is reaching the I/O limits of a disk subsystem or a disk. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
Describe the problem you're observing
Randomly (not at boot or pool import) system shows panic messages (see below) on the console. The system seemed to be operational. Upon trying to reboot I was unable to get a clean shutdown. Had several
Failed unmountingmessages with references to/rootand/var/log. Along with a failed to start Journal Service. Eventually it hung at asystemd-shutdown[1]: Syncing filesystems and block devices - timed out, issuing SIGKILL to PID ...Had to kill the power after waiting ~20 minutes.Upon reboot, everything seems fine. No issues with pool import,
zpool statuswas good. After an hour or so I would see the first panic message. Usually within 24 to 48 hours, had to reboot again.. it would not be a clean shutdown, required manual power off.The PANIC messages I have is very close to #13483 however, that discussion appears to be on pool import issues. I'm not having any issues with pool import so I'm documenting a new issue as the panic seems to have a different cause.
Include any warning/errors/backtraces from the system logs
NOTE: System is using ZFS Boot Menu, so this is ZFS on Root, but does not require the commonly used separate BOOT and ROOT pools. This method allows for a single ZFS pool (with encryption).
After reboot, status is good:
I tried to issue a
sudo zpool scrub k3s06and it hanged (did not return to prompt). I open another SSH session,zpool statusdoes not indicate a scrub is in progress. Checked the console and it had panic message scrolling on the screen.I did another unclean shutdown again and rebooted. Tried some of the suggestions within #13483
Then issues another scrub, which completed successfully:
As expected from the tunables, panic messages now became warning messages:
This seem to run clean as well:
This time I was able to get a clean shutdown. Rebooted without the tunables.
System is now running 3+ hours under normal load, no panic messages have been seen since yet. Will keep monitoring.
The text was updated successfully, but these errors were encountered: