linux-kernel - Re: 3.17-rc2 percpu-ref oops with virtblk remove

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <53FF7A3A.7060607@kernel.dk>
Date:	Thu, 28 Aug 2014 12:51:38 -0600
From:	Jens Axboe <axboe@...nel.dk>
To:	Will Deacon <will.deacon@....com>, rusty@...tcorp.com.au,
	tj@...nel.org, hch@....de
CC:	linux-kernel@...r.kernel.org
Subject: Re: 3.17-rc2 percpu-ref oops with virtblk remove

On 08/28/2014 12:50 PM, Will Deacon wrote:
> Hi guys,
> 
> I've been debugging an issue triggered by virtio-blk and vfio on an
> arm64 system running 3.17-rc2. The problem occurs when removing a
> virtio-pci block device, which triggers the following warning in the
> percpu-ref code:
> 
> WARNING: CPU: 0 PID: 1312 at lib/percpu-refcount.c:179 percpu_ref_kill_and_confirm+0x90/0x9c()
> percpu_ref_kill() called more than once!
> Modules linked in:
> CPU: 0 PID: 1312 Comm: vfio-setup.sh Not tainted 3.17.0-rc2+ #5
> Call trace:
> [<ffffffc0002b68b0>] percpu_ref_kill_and_confirm+0x8c/0x9c
> [<ffffffc0002938e8>] blk_mq_freeze_queue+0x34/0xc8
> [<ffffffc000294dac>] blk_mq_free_queue+0x1c/0x10c
> [<ffffffc00028cc1c>] blk_release_queue+0x68/0xa8
> [<ffffffc0002a7208>] kobject_release+0x40/0x84
> [<ffffffc0002a7284>] kobject_put+0x38/0x6c
> [<ffffffc000288414>] blk_put_queue+0xc/0x18
> [<ffffffc0002989b0>] disk_release+0x7c/0xb8
> [<ffffffc0003245f0>] device_release+0x30/0x98
> [<ffffffc0002a7208>] kobject_release+0x40/0x84
> [<ffffffc0002a7284>] kobject_put+0x38/0x6c
> [<ffffffc0002980cc>] put_disk+0x10/0x1c
> [<ffffffc00033b010>] virtblk_remove+0x74/0xd4
> 
> Some more debugging shows that virtblk_remove earlier called
> blk_cleanup_queue:
> 
> [<ffffffc0002b686c>] percpu_ref_kill_and_confirm+0x48/0x9c
> [<ffffffc0002938e8>] blk_mq_freeze_queue+0x34/0xc8
> [<ffffffc000288fd4>] blk_cleanup_queue+0x70/0xf4
> [<ffffffc00033afe0>] virtblk_remove+0x44/0xd4
> 
> which ends up marking the percpu ref as dead and queues some RCU work
> to switch over to the atomic_t version using call_rcu_sched. The
> virtblk_remove function then continues happily and calls put_disk (first
> backtrace above), which ends up trying to get exclusive access to the
> queue by killing the usage counter and waiting for the percpu reference
> to become zero. Unfortunately, since the reference is already marked as
> dead, the call to percpu_ref_kill triggers the warning. Worse still, we
> then queue the RCU callback a second time, which dies with something
> like:
> 
> Unable to handle kernel paging request at virtual address 87f971000
> pgd = ffffffc87aca8000
> [87f971000] *pgd=0000000000000000, *pud=0000000000000000
> Internal error: Oops: 96000005 [#1] PREEMPT SMP
> Modules linked in:
> CPU: 0 PID: 1312 Comm: systemd-udevd Tainted: G        W      3.17.0-rc2+ #9
> task: ffffffc87bb89480 ti: ffffffc87ace0000 task.ti: ffffffc87ace0000
> PC is at percpu_ref_kill_rcu+0x50/0x188
> LR is at percpu_ref_kill_rcu+0x6c/0x188
> pc : [<ffffffc0002b6a34>] lr : [<ffffffc0002b6a50>] pstate: 80000145
> sp : ffffffc87ace3840
> x29: ffffffc87ace3840 x28: ffffffc000470000 
> x27: 0000000000000000 x26: ffffffc87ff74a00 
> x25: ffffffc00061f238 x24: ffffffc07f858d68 
> x23: ffffffc000657000 x22: ffffffc07f858d88 
> x21: ffffffc00064bc68 x20: 0000000000000000 
> x19: 0000000000000000 x18: 0000007fc81227e0 
> x17: 0000007fb74f1194 x16: ffffffc000161a44 
> x15: 0000007fb757b5a0 x14: 0000000000000008 
> x13: 0000000000000000 x12: ffffffc000470c70 
> x11: 0000000000000005 x10: 0101010101010101 
> x9 : ffffffc000470c4b x8 : ffffffc000647318 
> x7 : fffffffffffffe40 x6 : 0000000000005b00 
> x5 : ffffffc00064bc68 x4 : 0000000000000038 
> x3 : 00000000000000ff x2 : 0000000000000000 
> x1 : ffffffc000657ee8 x0 : 000000087f971000 
> 
> because the percpu_ref has been freed from blk_mq_free_queue.
> 
> I'm not really sure how to fix this -- it seems like we shouldn't try to
> kill a reference that is already dead, but using __pcpu_ref_alive isn't
> the right answer. Simply removing the warning works for me (patch below),
> but that also feels like a hack (we skip the confirm callback, for a
> start).
> 
> Any ideas?

It's fixed in for-linus, which will go out soon, more stuff just kept
coming in. Fix:

http://git.kernel.dk/?p=linux-block.git;a=commit;h=cddd5d17642cc6881352732693c2ae6930e9ce65

-- 
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/