Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kernel panic during large files removal #583

Closed
mikhmv opened this issue Feb 24, 2012 · 3 comments
Closed

kernel panic during large files removal #583

mikhmv opened this issue Feb 24, 2012 · 3 comments
Milestone
0.6.0-rc7

Comments

@mikhmv
Copy link

mikhmv commented Feb 24, 2012

Hi, I have a blocking bug!!!

When I remove small files (40GB as an example) command rm is working well but on removing large files (184 GB) I have regularly kernel panic.

File:

 
-rwxr-x--- 1 oneadmin cloud 184G 2012-02-05 10:27 bioUML-DB-10003-disk-1.raw.bz2

Command:

$date; rm -r bioUML-DB-10003-disk-1.raw.bz2; date
Thu Feb 23 20:32:32 EST 2012
Thu Feb 23 20:42:01 EST 2012

After executing this command I got messages in dmesg.

$ uname -a
Linux s0 3.0.0-16-server #28-Ubuntu SMP Fri Jan 27 18:03:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ sudo zpool status
  pool: tank
 state: ONLINE
 scan: scrub repaired 0 in 7h41m with 0 errors on Wed Feb 15 17:42:42 2012
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        d1      ONLINE       0     0     0
        d2      ONLINE       0     0     0
        d3      ONLINE       0     0     0
        d4      ONLINE       0     0     0
        d5      ONLINE       0     0     0

errors: No known data errors

max@s0:~$ dpkg -s ubuntu-zfs
Package: ubuntu-zfs
Status: install ok installed
Priority: optional
Section: metapackages
Installed-Size: 32
Maintainer: Darik Horn 
Architecture: amd64
Version: 4~oneiric
Depends: spl, spl-dkms, zfs-dkms, zfsutils
Description: Native ZFS filesystem metapackage for Ubuntu.
max@s0:~$ dpkg -s zfsutils
Package: zfsutils
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 876
Maintainer: Darik Horn 
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.34-0ubuntu1~oneiric1
Replaces: zfs
Depends: libc6 (>= 2.8), libefi1, libnvpair1, libspl1, libuuid1 (>= 2.16), libuutil1, libzavl1, libzfs1, libzpool1
Recommends: zfs-dkms
Suggests: nfs-kernel-server, zfs-initramfs
Conflicts: zfs, zfs-fuse
Conffiles:
 /etc/default/zfs df1821c2e84e8f19ebc805f3a656cf1c
 /etc/init.d/zfs d549d0455e2299b4318cd9038742ec4a
 /etc/zfs/zdev.conf b006284e64b215ca619aeb56d2df9bf5
Description: Native ZFS management utilities for Linux
 This package provides the zpool and zfs commands that are used to
 manage ZFS filesystems.

dmesg:

[687848.632630] INFO: task oned:8231 blocked for more than 120 seconds.
[687848.651278] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.688812] oned            D 0000000000000038     0  8231      1 0x00000000
[687848.688826]  ffff880aed60db88 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.688837]  ffff880aed60dfd8 ffff880aed60dfd8 ffff880aed60dfd8 0000000000012a40
[687848.688845]  ffff8827b1611720 ffff880e94312e40 ffff880aed60db98 ffff8827f7dd3b30
[687848.688852] Call Trace:
[687848.688870]  [] schedule+0x3f/0x60
[687848.688902]  [] cv_wait_common+0x77/0xd0 [spl]
[687848.688911]  [] ? add_wait_queue+0x60/0x60
[687848.688922]  [] __cv_wait+0x13/0x20 [spl]
[687848.688982]  [] txg_wait_open+0x73/0xa0 [zfs]
[687848.689009]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.689046]  [] zfs_write+0x377/0xc50 [zfs]
[687848.689056]  [] ? __dentry_open+0x2bb/0x320
[687848.689063]  [] ? vma_adjust+0x464/0x490
[687848.689096]  [] ? dmu_object_size_from_db+0x69/0xa0 [zfs]
[687848.689132]  [] zpl_write_common+0x52/0x80 [zfs]
[687848.689165]  [] zpl_write+0x68/0xa0 [zfs]
[687848.689182]  [] vfs_write+0xb3/0x180
[687848.689201]  [] sys_write+0x4a/0x90
[687848.689223]  [] system_call_fastpath+0x16/0x1b
[687848.689245] INFO: task kvm:8147 blocked for more than 120 seconds.
[687848.708632] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.748404] kvm             D 0000000000000012     0  8147      1 0x00000000
[687848.748418]  ffff88396b3e3a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.748428]  ffff88396b3e3fd8 ffff88396b3e3fd8 ffff88396b3e3fd8 0000000000012a40
[687848.748439]  ffff8827b1ac5c80 ffff883c93af9720 ffff88396b3e3a88 ffff8827f7dd3b30
[687848.748450] Call Trace:
[687848.748465]  [] schedule+0x3f/0x60
[687848.748480]  [] cv_wait_common+0x77/0xd0 [spl]
[687848.748488]  [] ? add_wait_queue+0x60/0x60
[687848.748503]  [] __cv_wait+0x13/0x20 [spl]
[687848.748547]  [] txg_wait_open+0x73/0xa0 [zfs]
[687848.748578]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.748611]  [] zfs_write+0x377/0xc50 [zfs]
[687848.748621]  [] ? __perf_event_task_sched_out+0x30/0x60
[687848.748630]  [] ? rb_erase+0xce/0x150
[687848.748638]  [] ? __remove_hrtimer+0x60/0xc0
[687848.748671]  [] zpl_write_common+0x52/0x80 [zfs]
[687848.748703]  [] zpl_write+0x68/0xa0 [zfs]
[687848.748740]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687848.748749]  [] do_loop_readv_writev+0x59/0x90
[687848.748758]  [] do_readv_writev+0x1ae/0x1d0
[687848.748769]  [] ? do_futex+0xa8/0x1d0
[687848.748777]  [] vfs_writev+0x3c/0x50
[687848.748784]  [] sys_pwritev+0xba/0xc0
[687848.748795]  [] system_call_fastpath+0x16/0x1b
[687848.748800] INFO: task kvm:8242 blocked for more than 120 seconds.
[687848.769694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.811872] kvm             D 0000000000000038     0  8242      1 0x00000000
[687848.811886]  ffff880ab95e1b88 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.811897]  ffff880ab95e1fd8 ffff880ab95e1fd8 ffff880ab95e1fd8 0000000000012a40
[687848.811906]  ffff8827b1611720 ffff880de3092e40 ffff880ab95e1b98 ffff8827f7dd3b30
[687848.811915] Call Trace:
[687848.811924]  [] schedule+0x3f/0x60
[687848.811938]  [] cv_wait_common+0x77/0xd0 [spl]
[687848.811947]  [] ? add_wait_queue+0x60/0x60
[687848.811960]  [] __cv_wait+0x13/0x20 [spl]
[687848.811994]  [] txg_wait_open+0x73/0xa0 [zfs]
[687848.812036]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.812071]  [] zfs_write+0x377/0xc50 [zfs]
[687848.812105]  [] zpl_write_common+0x52/0x80 [zfs]
[687848.812138]  [] zpl_write+0x68/0xa0 [zfs]
[687848.812148]  [] vfs_write+0xb3/0x180
[687848.812154]  [] sys_pwrite64+0xa2/0xb0
[687848.812163]  [] system_call_fastpath+0x16/0x1b
[687848.812169] INFO: task kvm:8243 blocked for more than 120 seconds.
[687848.834606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.879674] kvm             D 0000000000000002     0  8243      1 0x00000000
[687848.879683]  ffff880b9dc97b88 0000000000000086 ffff880b9dc97b28 ffffffffa01052b6
[687848.879694]  ffff880b9dc97fd8 ffff880b9dc97fd8 ffff880b9dc97fd8 0000000000012a40
[687848.879705]  ffff8827b28dc560 ffff880de3091720 ffff880b9dc97b98 ffff8827f7dd3b30
[687848.879713] Call Trace:
[687848.879727]  [] ? kmem_free_debug+0x16/0x20 [spl]
[687848.879736]  [] schedule+0x3f/0x60
[687848.879748]  [] cv_wait_common+0x77/0xd0 [spl]
[687848.879763]  [] ? add_wait_queue+0x60/0x60
[687848.879789]  [] __cv_wait+0x13/0x20 [spl]
[687848.879842]  [] txg_wait_open+0x73/0xa0 [zfs]
[687848.879885]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.879929]  [] zfs_write+0x377/0xc50 [zfs]
[687848.879972]  [] zpl_write_common+0x52/0x80 [zfs]
[687848.880018]  [] zpl_write+0x68/0xa0 [zfs]
[687848.880058]  [] vfs_write+0xb3/0x180
[687848.880074]  [] sys_pwrite64+0xa2/0xb0
[687848.880096]  [] system_call_fastpath+0x16/0x1b
[687848.880113] INFO: task kvm:11994 blocked for more than 120 seconds.
[687848.903888] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.951632] kvm             D 0000000000000038     0 11994      1 0x00000000
[687848.951640]  ffff883fda8a1a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.951650]  ffff883fda8a1fd8 ffff883fda8a1fd8 ffff883fda8a1fd8 0000000000012a40
[687848.951660]  ffff883fdca02e40 ffff883fda82dc80 ffff883fda8a1a88 ffff8827f7dd3b30
[687848.951671] Call Trace:
[687848.951678]  [] schedule+0x3f/0x60
[687848.951692]  [] cv_wait_common+0x77/0xd0 [spl]
[687848.951698]  [] ? add_wait_queue+0x60/0x60
[687848.951711]  [] __cv_wait+0x13/0x20 [spl]
[687848.951746]  [] txg_wait_open+0x73/0xa0 [zfs]
[687848.951773]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.951806]  [] zfs_write+0x377/0xc50 [zfs]
[687848.951835]  [] ? kvm_fast_pio_out+0x64/0x80 [kvm]
[687848.951846]  [] ? handle_exit+0x163/0x380 [kvm_amd]
[687848.951880]  [] zpl_write_common+0x52/0x80 [zfs]
[687848.951912]  [] zpl_write+0x68/0xa0 [zfs]
[687848.951930]  [] ? svm_get_msr+0x155/0x320 [kvm_amd]
[687848.951973]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687848.951988]  [] do_loop_readv_writev+0x59/0x90
[687848.952029]  [] do_readv_writev+0x1ae/0x1d0
[687848.952056]  [] vfs_writev+0x3c/0x50
[687848.952078]  [] sys_pwritev+0xba/0xc0
[687848.952094]  [] system_call_fastpath+0x16/0x1b
[687848.952108] INFO: task kvm:8183 blocked for more than 120 seconds.
[687848.976763] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.026272] kvm             D 0000000000000026     0  8183      1 0x00000000
[687849.026280]  ffff8806b2143a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687849.026290]  ffff8806b2143fd8 ffff8806b2143fd8 ffff8806b2143fd8 0000000000012a40
[687849.026299]  ffff883ff157dc80 ffff8807f55d0000 ffff8806b2143a88 ffff8827f7dd3b30
[687849.026308] Call Trace:
[687849.026315]  [] schedule+0x3f/0x60
[687849.026327]  [] cv_wait_common+0x77/0xd0 [spl]
[687849.026334]  [] ? add_wait_queue+0x60/0x60
[687849.026347]  [] __cv_wait+0x13/0x20 [spl]
[687849.026381]  [] txg_wait_open+0x73/0xa0 [zfs]
[687849.026409]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.026446]  [] zfs_write+0x377/0xc50 [zfs]
[687849.026454]  [] ? __perf_event_task_sched_out+0x30/0x60
[687849.026489]  [] zpl_write_common+0x52/0x80 [zfs]
[687849.026522]  [] zpl_write+0x68/0xa0 [zfs]
[687849.026529]  [] ? rw_copy_check_uvector+0xf5/0x130
[687849.026564]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.026571]  [] do_loop_readv_writev+0x59/0x90
[687849.026577]  [] do_readv_writev+0x1ae/0x1d0
[687849.026584]  [] ? futex_wake+0x113/0x130
[687849.026592]  [] ? do_futex+0xa8/0x1d0
[687849.026597]  [] vfs_writev+0x3c/0x50
[687849.026605]  [] sys_pwritev+0xba/0xc0
[687849.026615]  [] system_call_fastpath+0x16/0x1b
[687849.026630] INFO: task kvm:8240 blocked for more than 120 seconds.
[687849.052708] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.105106] kvm             D 0000000000000038     0  8240      1 0x00000000
[687849.105118]  ffff883459339a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.105127]  ffff883459339fd8 ffff883459339fd8 ffff883459339fd8 0000000000012a40
[687849.105137]  ffff8827b1611720 ffff8837f7134560 ffff883459339a88 ffff8827f7dd3b30
[687849.105147] Call Trace:
[687849.105154]  [] schedule+0x3f/0x60
[687849.105168]  [] cv_wait_common+0x77/0xd0 [spl]
[687849.105175]  [] ? add_wait_queue+0x60/0x60
[687849.105191]  [] __cv_wait+0x13/0x20 [spl]
[687849.105228]  [] txg_wait_open+0x73/0xa0 [zfs]
[687849.105260]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.105294]  [] zfs_write+0x377/0xc50 [zfs]
[687849.105333]  [] zpl_write_common+0x52/0x80 [zfs]
[687849.105366]  [] zpl_write+0x68/0xa0 [zfs]
[687849.105400]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.105408]  [] do_loop_readv_writev+0x59/0x90
[687849.105416]  [] do_readv_writev+0x1ae/0x1d0
[687849.105423]  [] vfs_writev+0x3c/0x50
[687849.105427]  [] sys_pwritev+0xba/0xc0
[687849.105433]  [] system_call_fastpath+0x16/0x1b
[687849.105438] INFO: task kvm:8244 blocked for more than 120 seconds.
[687849.131994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.184830] kvm             D 000000000000003b     0  8244      1 0x00000000
[687849.184840]  ffff8833a1c2db88 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.184850]  ffff8833a1c2dfd8 ffff8833a1c2dfd8 ffff8833a1c2dfd8 0000000000012a40
[687849.184861]  ffff8827b1615c80 ffff8837f7135c80 ffff8833a1c2db98 ffff8827f7dd3b30
[687849.184870] Call Trace:
[687849.184878]  [] schedule+0x3f/0x60
[687849.184890]  [] cv_wait_common+0x77/0xd0 [spl]
[687849.184899]  [] ? add_wait_queue+0x60/0x60
[687849.184912]  [] __cv_wait+0x13/0x20 [spl]
[687849.184947]  [] txg_wait_open+0x73/0xa0 [zfs]
[687849.184975]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.185007]  [] zfs_write+0x377/0xc50 [zfs]
[687849.185016]  [] ? _raw_spin_lock+0xe/0x20
[687849.185025]  [] ? fsnotify_clear_marks_by_inode+0x94/0xf0
[687849.185036]  [] ? __perf_event_task_sched_out+0x30/0x60
[687849.185045]  [] ? hrtick_update+0x38/0x40
[687849.185090]  [] zpl_write_common+0x52/0x80 [zfs]
[687849.185131]  [] zpl_write+0x68/0xa0 [zfs]
[687849.185144]  [] vfs_write+0xb3/0x180
[687849.185159]  [] sys_pwrite64+0xa2/0xb0
[687849.185176]  [] system_call_fastpath+0x16/0x1b
[687849.185186] INFO: task kvm:8246 blocked for more than 120 seconds.
[687849.212468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.267657] kvm             D 000000000000003b     0  8246      1 0x00000000
[687849.267665]  ffff883799351a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.267676]  ffff883799351fd8 ffff883799351fd8 ffff883799351fd8 0000000000012a40
[687849.267686]  ffff8827b1615c80 ffff88362dd2dc80 ffff883799351a88 ffff8827f7dd3b30
[687849.267693] Call Trace:
[687849.267699]  [] schedule+0x3f/0x60
[687849.267710]  [] cv_wait_common+0x77/0xd0 [spl]
[687849.267716]  [] ? add_wait_queue+0x60/0x60
[687849.267727]  [] __cv_wait+0x13/0x20 [spl]
[687849.267764]  [] txg_wait_open+0x73/0xa0 [zfs]
[687849.267791]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.267823]  [] zfs_write+0x377/0xc50 [zfs]
[687849.267857]  [] zpl_write_common+0x52/0x80 [zfs]
[687849.267900]  [] zpl_write+0x68/0xa0 [zfs]
[687849.267939]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.267953]  [] do_loop_readv_writev+0x59/0x90
[687849.267968]  [] do_readv_writev+0x1ae/0x1d0
[687849.267981]  [] vfs_writev+0x3c/0x50
[687849.267996]  [] sys_pwritev+0xba/0xc0
[687849.268011]  [] system_call_fastpath+0x16/0x1b
[687849.268039] INFO: task kvm:8241 blocked for more than 120 seconds.
[687849.296215] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.353700] kvm             D 0000000000000038     0  8241      1 0x00000000
[687849.353709]  ffff8832fe875a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.353731]  ffff8832fe875fd8 ffff8832fe875fd8 ffff8832fe875fd8 0000000000012a40
[687849.353771]  ffff8827b1611720 ffff883653405c80 ffff8832fe875a88 ffff8827f7dd3b30
[687849.353795] Call Trace:
[687849.353809]  [] schedule+0x3f/0x60
[687849.353841]  [] cv_wait_common+0x77/0xd0 [spl]
[687849.353859]  [] ? add_wait_queue+0x60/0x60
[687849.353886]  [] __cv_wait+0x13/0x20 [spl]
[687849.353926]  [] txg_wait_open+0x73/0xa0 [zfs]
[687849.353973]  [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.354027]  [] zfs_write+0x377/0xc50 [zfs]
[687849.354047]  [] ? _raw_spin_lock+0xe/0x20
[687849.354092]  [] zpl_write_common+0x52/0x80 [zfs]
[687849.354133]  [] zpl_write+0x68/0xa0 [zfs]
[687849.354146]  [] ? perf_event_task_sched_out+0x2e/0xa0
[687849.354184]  [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.354202]  [] do_loop_readv_writev+0x59/0x90
[687849.354217]  [] do_readv_writev+0x1ae/0x1d0
[687849.354229]  [] vfs_writev+0x3c/0x50
[687849.354240]  [] sys_pwritev+0xba/0xc0
[687849.354252]  [] system_call_fastpath+0x16/0x1b
@mikhmv
Copy link
Author

mikhmv commented Feb 24, 2012

Sorry for a strange formatting in my bug report. I don't know how it happened and how to fix it

@mikhmv mikhmv mentioned this issue Feb 24, 2012
Closed
@behlendorf
Copy link
Contributor

What your seeing isn't a panic it's just a warning about the slow unlinks. As for the bug itself, this issue was recently fixed in the daily ppa (or the latest master source from github). I'd suggest updating your system to that version of the code which contains quite a few other improvements as well. Please, let us know if that doesn't resolve the issue.

@mikhmv
Copy link
Author

mikhmv commented Mar 5, 2012

Hi Behlendorf,

It resolved the problem. Updated version work great.

Thanks

@mikhmv mikhmv closed this as completed Mar 5, 2012
pcd1193182 pushed a commit to pcd1193182/zfs that referenced this issue Sep 26, 2023
@delphix-devops-bot
Merge pull request openzfs#583 from delphix/projects/merge-upstream/m…
5fe75a3 
…aster

Merge remote-tracking branch '6.0/stage' into 'master'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.6.0-rc7
Development

No branches or pull requests
2 participants
@behlendorf @mikhmv