Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

race window in zvol_open() resulting in use after free, zv_name modified without checking zv_open_count, deadlock between zvol_open() and __zvol_create_minor() #4344

Closed
bprotopopov opened this issue Feb 18, 2016 · 2 comments
Milestone

Comments

@bprotopopov
Copy link
Contributor

A pointer to zvol_state_t for a given zvol is stored in bdev->bd_disk->private_data so that it can be used in various zvol device methods. It is assumed that such use occurs in the context of an open file descriptor referencing the zvol in question, and the corresponding reference count (zv_open_count) is maintained in the zvol_state_t structure.

This allows for a simple way of knowing when it is and it not OK to free that structure in zvol_free() - one can check the zv_open_count under zvol_state_lock, and if it is positive, the structure is in use, and should be left alone; zvol_remove_minors() should follow this rule.

There is an issue with the code path in zvol_open() when zv_open_count is zero on entry to the function. The private_data is saved in the local variable while not under protection of zvol_state_lock, and only then the lock is taken. This creates a small race window for zvol_remove_minors() to come in take the lock, find the zv_open_count equal to zero, and call zvol_free(), leaving the zvol_open() code path to experience the "use after free" situation.

Function zvol_rename_minors() does not check the reference count before modifying the zv_name, which is not good since zvol_ioctl(), for instance, accesses zv_name without holding zvol_state_lock (but with zv_open_count being positive).

Function __zvol_create_minor() calls add_disk() while holding zvol_state_lock, which results in a deadlock with zvol_open() if both are invoked concurrently.

@bprotopopov bprotopopov changed the title race window in zvol_open() resulting in use after free, zv_name modified without checking zv_open_count race window in zvol_open() resulting in use after free, zv_name modified without checking zv_open_count, deadlock between zvol_open() and __zvol_create_minor() Feb 18, 2016
@bprotopopov
Copy link
Contributor Author

The work is done on branch issue-4344

@bprotopopov
Copy link
Contributor Author

Tested as follows.

[root@centos-6 ~]# /usr/share/zfs/zconfig.sh -c -v -s10
Destroying 
1    persistent zpool.cache             Pass
2    scan disks for pools to import     Pass
3    zpool import/export device         Pass
4    zpool insmod/rmmod device          Pass
5    zvol+ext2 volume                   Pass
6    zvol+ext2 snapshot                 Pass
7    zvol+ext2 clone                    Pass
8    zfs send/receive                   Pass
9    zpool events                       Pass
10   zpool add/remove vdev              Skip
[root@centos-6 ~]# zpool import pool1
The ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them.
[root@centos-6 ~]# modprobe zfs
[root@centos-6 ~]# zpool import pool1
[root@centos-6 ~]# zfs list -r -t all
NAME                       USED  AVAIL  REFER  MOUNTPOINT
pool1            722K   975M    19K  /pool1
pool1/zvol         8K   975M     8K  -
pool1/zvol@1        0      -     8K  -
pool1/zvol@2        0      -     8K  -
pool1/zvol@3        0      -     8K  -
pool1/zvol@4        0      -     8K  -
pool1/zvol@5        0      -     8K  -
pool1/zvol@6        0      -     8K  -
pool1/zvol@7        0      -     8K  -
pool1/zvol@8        0      -     8K  -
pool1/zvol@9        0      -     8K  -
pool1/zvol@10       0      -     8K  -
pool1/zvol@11       0      -     8K  -
pool1/zvol@12       0      -     8K  -
pool1/zvol@13       0      -     8K  -
pool1/zvol@14       0      -     8K  -
pool1/zvol@15       0      -     8K  -
pool1/zvol@16       0      -     8K  -
pool1/zvol@17       0      -     8K  -
pool1/zvol@18       0      -     8K  -
pool1/zvol@19       0      -     8K  -
pool1/zvol@20       0      -     8K  -
pool1/zvol@21       0      -     8K  -
pool1/zvol@22       0      -     8K  -
pool1/zvol@23       0      -     8K  -
pool1/zvol@24       0      -     8K  -
pool1/zvol@25       0      -     8K  -
pool1/zvol@26       0      -     8K  -
pool1/zvol@27       0      -     8K  -
pool1/zvol@28       0      -     8K  -
pool1/zvol@29       0      -     8K  -
pool1/zvol@30       0      -     8K  -
pool1/zvol@31       0      -     8K  -
pool1/zvol@32       0      -     8K  -
pool1/zvol@33       0      -     8K  -
pool1/zvol@34       0      -     8K  -
pool1/zvol@35       0      -     8K  -
pool1/zvol@36       0      -     8K  -
pool1/zvol@37       0      -     8K  -
pool1/zvol@38       0      -     8K  -
pool1/zvol@39       0      -     8K  -
pool1/zvol@40       0      -     8K  -
pool1/zvol@41       0      -     8K  -
pool1/zvol@42       0      -     8K  -
pool1/zvol@43       0      -     8K  -
pool1/zvol@44       0      -     8K  -
pool1/zvol@45       0      -     8K  -
pool1/zvol@46       0      -     8K  -
pool1/zvol@47       0      -     8K  -
pool1/zvol@48       0      -     8K  -
pool1/zvol@49       0      -     8K  -
pool1/zvol@50       0      -     8K  -
pool1/zvol@51       0      -     8K  -
pool1/zvol@52       0      -     8K  -
pool1/zvol@53       0      -     8K  -
pool1/zvol@54       0      -     8K  -
pool1/zvol@55       0      -     8K  -
pool1/zvol@56       0      -     8K  -
pool1/zvol@57       0      -     8K  -
pool1/zvol@58       0      -     8K  -
pool1/zvol@59       0      -     8K  -
pool1/zvol@60       0      -     8K  -
pool1/zvol@61       0      -     8K  -
pool1/zvol@62       0      -     8K  -
pool1/zvol@63       0      -     8K  -
pool1/zvol@64       0      -     8K  -
pool1/zvol@65       0      -     8K  -
pool1/zvol@66       0      -     8K  -
pool1/zvol@67       0      -     8K  -
pool1/zvol@68       0      -     8K  -
pool1/zvol@69       0      -     8K  -
pool1/zvol@70       0      -     8K  -
pool1/zvol@71       0      -     8K  -
pool1/zvol@72       0      -     8K  -
pool1/zvol@73       0      -     8K  -
pool1/zvol@74       0      -     8K  -
pool1/zvol@75       0      -     8K  -
pool1/zvol@76       0      -     8K  -
pool1/zvol@77       0      -     8K  -
pool1/zvol@78       0      -     8K  -
pool1/zvol@79       0      -     8K  -
pool1/zvol@80       0      -     8K  -
pool1/zvol@81       0      -     8K  -
pool1/zvol@82       0      -     8K  -
pool1/zvol@83       0      -     8K  -
pool1/zvol@84       0      -     8K  -
pool1/zvol@85       0      -     8K  -
pool1/zvol@86       0      -     8K  -
pool1/zvol@87       0      -     8K  -
pool1/zvol@88       0      -     8K  -
pool1/zvol@89       0      -     8K  -
pool1/zvol@90       0      -     8K  -
pool1/zvol@91       0      -     8K  -
pool1/zvol@92       0      -     8K  -
pool1/zvol@93       0      -     8K  -
pool1/zvol@94       0      -     8K  -
pool1/zvol@95       0      -     8K  -
pool1/zvol@96       0      -     8K  -
pool1/zvol@97       0      -     8K  -
pool1/zvol@98       0      -     8K  -
pool1/zvol@99       0      -     8K  -
pool1/zvol@100      0      -     8K  -
[root@centos-6 ~]# for t in $(seq 1 10); do echo " --- Test $t ---"; for (( i=0; i<10; i++ )); do echo "Iteration $i";     zfs set snapdev=visible pool1/zvol; zfs set snapdev=hidden pool1/zvol; done; done
 --- Test 1 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 2 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 3 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 4 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 5 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 6 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 7 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 8 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 9 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
 --- Test 10 ---
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
[root@centos-6 ~]# zpool export pool1
[root@centos-6 ~]# modprobe -r zfs
[root@centos-6 ~]# modprobe zfs
[root@centos-6 ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@centos-6 ~]# cat ~/code/tests/zvol_minor.bash
#!/bin/bash

# Test the following concurrently in more than one pool:
# - create zvols, set snapev=visible, create snapshots thereof
# - flip the spabdev visible/hidden a few times
# - destroy zvols and their snapshots with snapdev=visible


function zvol_ops()
{
    # zvol name (poolname/zvol), initial value of snapdev, number of snaps
    # number of iterations for changing snapdev=visible/hidden
    local zvol=$1
    local snapdev_init=$2
    local nsnaps=$3
    local iterations=$4

    echo "zfs create -V1G -s -b 64k -o snapdev=\"$snapdev_init\" $zvol"
    zfs create -V1G -s -b 64k -o snapdev="$snapdev_init" $zvol

    for (( i=0; i<$iterations; i++ )) {
    echo "    zfs snapshot ${zvol}@\"$i\""
    zfs snapshot ${zvol}@"$i"
    }

    local snapdev="$snapdev_init"

    for (( i=0; i<$iterations; i++ )) {
    if [ "$snapdev" = "visible" ]; then
        snapdev="hidden";
    else
        snapdev="visible";
        fi
        echo "          zfs set snapdev=\"$snapdev\" $zvol"
        zfs set snapdev="$snapdev" $zvol
    }

    echo "    zfs set snapdev=\"$snapdev_init\" $zvol"
    zfs set snapdev="$snapdev_init" $zvol
    echo "zfs destroy $zvol"
    zfs destroy -r $zvol
}

# main script
snapdev_start="visible"
for pool in "pool1" "pool2"; do
    ( echo "zpool import $pool"; zpool import $pool;
    for (( i=1; i<=10; i++ )) {
    zvol_ops "${pool}/zvol$i" "$snapdev_start" 10 1 &

    if [ "$snapdev_start" = "visible" ]; then
        snapdev_start="hidden";
    else
        snapdev_start="visible";
        fi
    }; wait; echo "zpool export $pool"; zpool export $pool ) &
done

wait
[root@centos-6 ~]# 
[root@centos-6 ~]# ~/code/tests/zvol_minor.bash
zpool import pool1
zpool import pool2
zfs create -V1G -s -b 64k -o snapdev="visible" pool2/zvol1
zfs create -V1G -s -b 64k -o snapdev="hidden" pool2/zvol2
zfs create -V1G -s -b 64k -o snapdev="visible" pool2/zvol3
zfs create -V1G -s -b 64k -o snapdev="visible" pool2/zvol5
zfs create -V1G -s -b 64k -o snapdev="hidden" pool2/zvol8
zfs create -V1G -s -b 64k -o snapdev="visible" pool2/zvol7
zfs create -V1G -s -b 64k -o snapdev="hidden" pool2/zvol4
zfs create -V1G -s -b 64k -o snapdev="hidden" pool2/zvol10
zfs create -V1G -s -b 64k -o snapdev="visible" pool2/zvol9
zfs create -V1G -s -b 64k -o snapdev="hidden" pool2/zvol6
zfs create -V1G -s -b 64k -o snapdev="visible" pool1/zvol1
zfs create -V1G -s -b 64k -o snapdev="hidden" pool1/zvol2
zfs create -V1G -s -b 64k -o snapdev="visible" pool1/zvol3
zfs create -V1G -s -b 64k -o snapdev="hidden" pool1/zvol4
zfs create -V1G -s -b 64k -o snapdev="visible" pool1/zvol5
zfs create -V1G -s -b 64k -o snapdev="hidden" pool1/zvol8
zfs create -V1G -s -b 64k -o snapdev="hidden" pool1/zvol6
zfs create -V1G -s -b 64k -o snapdev="visible" pool1/zvol7
zfs create -V1G -s -b 64k -o snapdev="visible" pool1/zvol9
zfs create -V1G -s -b 64k -o snapdev="hidden" pool1/zvol10
    zfs snapshot pool2/zvol2@"0"
    zfs snapshot pool2/zvol8@"0"
    zfs snapshot pool2/zvol4@"0"
    zfs snapshot pool2/zvol10@"0"
    zfs snapshot pool2/zvol9@"0"
    zfs snapshot pool2/zvol1@"0"
    zfs snapshot pool2/zvol5@"0"
    zfs snapshot pool2/zvol7@"0"
    zfs snapshot pool2/zvol3@"0"
    zfs snapshot pool2/zvol6@"0"
          zfs set snapdev="visible" pool2/zvol2
    zfs snapshot pool1/zvol1@"0"
          zfs set snapdev="visible" pool2/zvol10
          zfs set snapdev="hidden" pool2/zvol5
          zfs set snapdev="visible" pool2/zvol8
          zfs set snapdev="hidden" pool2/zvol3
          zfs set snapdev="visible" pool2/zvol6
          zfs set snapdev="hidden" pool2/zvol7
          zfs set snapdev="hidden" pool2/zvol1
          zfs set snapdev="visible" pool2/zvol4
          zfs set snapdev="hidden" pool2/zvol9
    zfs snapshot pool1/zvol10@"0"
    zfs snapshot pool1/zvol5@"0"
    zfs snapshot pool1/zvol4@"0"
    zfs snapshot pool1/zvol8@"0"
    zfs snapshot pool1/zvol9@"0"
    zfs snapshot pool1/zvol6@"0"
    zfs snapshot pool1/zvol7@"0"
    zfs snapshot pool1/zvol2@"0"
    zfs snapshot pool1/zvol3@"0"
    zfs set snapdev="hidden" pool2/zvol2
          zfs set snapdev="hidden" pool1/zvol1
    zfs set snapdev="hidden" pool2/zvol10
    zfs set snapdev="hidden" pool2/zvol4
    zfs set snapdev="visible" pool2/zvol5
    zfs set snapdev="hidden" pool2/zvol8
    zfs set snapdev="visible" pool2/zvol9
    zfs set snapdev="visible" pool2/zvol3
    zfs set snapdev="visible" pool2/zvol1
    zfs set snapdev="visible" pool2/zvol7
    zfs set snapdev="hidden" pool2/zvol6
          zfs set snapdev="visible" pool1/zvol8
zfs destroy pool2/zvol2
          zfs set snapdev="hidden" pool1/zvol7
          zfs set snapdev="hidden" pool1/zvol3
          zfs set snapdev="hidden" pool1/zvol5
    zfs set snapdev="visible" pool1/zvol1
          zfs set snapdev="hidden" pool1/zvol9
          zfs set snapdev="visible" pool1/zvol4
          zfs set snapdev="visible" pool1/zvol6
          zfs set snapdev="visible" pool1/zvol2
          zfs set snapdev="visible" pool1/zvol10
zfs destroy pool2/zvol8
zfs destroy pool2/zvol4
zfs destroy pool2/zvol10
zfs destroy pool2/zvol3
zfs destroy pool2/zvol5
zfs destroy pool2/zvol9
zfs destroy pool2/zvol7
zfs destroy pool2/zvol6
zfs destroy pool2/zvol1
    zfs set snapdev="hidden" pool1/zvol8
    zfs set snapdev="visible" pool1/zvol7
    zfs set snapdev="visible" pool1/zvol9
    zfs set snapdev="visible" pool1/zvol3
    zfs set snapdev="hidden" pool1/zvol2
    zfs set snapdev="hidden" pool1/zvol10
    zfs set snapdev="visible" pool1/zvol5
    zfs set snapdev="hidden" pool1/zvol6
zfs destroy pool1/zvol1
    zfs set snapdev="hidden" pool1/zvol4
zpool export pool2
zfs destroy pool1/zvol8
zfs destroy pool1/zvol7
zfs destroy pool1/zvol10
zfs destroy pool1/zvol5
zfs destroy pool1/zvol2
zfs destroy pool1/zvol9
zfs destroy pool1/zvol4
zfs destroy pool1/zvol6
zfs destroy pool1/zvol3
zpool export pool1
[root@centos-6 ~]# modprobe -r zfs
[root@centos-6 ~]#

behlendorf pushed a commit to behlendorf/zfs that referenced this issue Feb 22, 2016
Close the race window in zvol_open() to prevent removal of
zvol_state in the 'first open' code path. Move the call to
check_disk_change() under zvol_state_lock to make sure the
zvol_media_changed() and zvol_revalidate_disk() called by
check_disk_change() are invoked with positive zv_open_count.

Skip opened zvols when removing minors and set private_data
to NULL for zvols that are not in use whose minors are being
removed, to indicate to zvol_open() that the state is gone.
Skip opened zvols when renaming minors to avoid modifying
zv_name that might be in use, e.g. in zvol_ioctl().

Drop zvol_state_lock before calling add_disk() when creating
minors to avoid deadlocks with zvol_open().

Wrap dmu_objset_find() with spl_fstran_mark()/unmark().

Signed-off-by: Boris Protopopov <boris.protopopov@actifio.com>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#4344
nedbass pushed a commit that referenced this issue Mar 23, 2016
Close the race window in zvol_open() to prevent removal of
zvol_state in the 'first open' code path. Move the call to
check_disk_change() under zvol_state_lock to make sure the
zvol_media_changed() and zvol_revalidate_disk() called by
check_disk_change() are invoked with positive zv_open_count.

Skip opened zvols when removing minors and set private_data
to NULL for zvols that are not in use whose minors are being
removed, to indicate to zvol_open() that the state is gone.
Skip opened zvols when renaming minors to avoid modifying
zv_name that might be in use, e.g. in zvol_ioctl().

Drop zvol_state_lock before calling add_disk() when creating
minors to avoid deadlocks with zvol_open().

Wrap dmu_objset_find() with spl_fstran_mark()/unmark().

Signed-off-by: Boris Protopopov <boris.protopopov@actifio.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Closes #4344
@behlendorf behlendorf added this to the 0.6.5.6 milestone Mar 23, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants