linux-kernel - RE: [PATCH V1] block: Fix null pointer dereference issue on struct io

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7e32c001d222483394327c5802a6c985@quicinc.com>
Date:   Thu, 18 May 2023 09:25:52 +0000
From:   "Pradeep Pragallapati (QUIC)" <quic_pragalla@...cinc.com>
To:     Damien Le Moal <dlemoal@...nel.org>
CC:     Yu Kuai <yukuai1@...weicloud.com>,
        "axboe@...nel.dk" <axboe@...nel.dk>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "yukuai (C)" <yukuai3@...wei.com>,
        Christoph Hellwig <hch@...radead.org>
Subject: RE: [PATCH V1] block: Fix null pointer dereference issue on struct
 io_cq

Hi,

-----Original Message-----
From: Damien Le Moal <dlemoal@...nel.org> 
Sent: Wednesday, May 17, 2023 3:02 PM
To: Christoph Hellwig <hch@...radead.org>
Cc: Yu Kuai <yukuai1@...weicloud.com>; Pradeep Pragallapati (QUIC) <quic_pragalla@...cinc.com>; axboe@...nel.dk; linux-block@...r.kernel.org; linux-kernel@...r.kernel.org; yukuai (C) <yukuai3@...wei.com>
Subject: Re: [PATCH V1] block: Fix null pointer dereference issue on struct io_cq

On 5/17/23 18:21, Christoph Hellwig wrote:
> On Wed, May 17, 2023 at 06:20:19PM +0900, Damien Le Moal wrote:
>> twice for the same icq. The missing rcu lock in ioc_exit_icqs() 
>> already was in itself a bug, and the missing flag check is another.
> 
> spinlocks imply a rcu critical section, no need to duplicate it.

Right. And I misread the code. As Yu said, given that ioc_exit_icqs() iterates the list of icqs under ioc->lock and the ioc is removed from that list under the same lock, ioc_exit_icqs() should never see an icq that went through ioc_destroy_icq()...
Very weird.

This weird can be possible 
1. updating icq_hint which is annotated as __rcu type without RCU-protected context in ioc_destroy_icq().
Moreover, this was taken care in else part of ioc_release_fn() by rcu_read_lock/unlock() but missed in if statement which can lead to this weird.

2. extracting icq from hlist/list elements are done using rcu locks protected in ioc_clear_queue() but same was not at ioc_exit_icqs().

So, far we have seen 10+ instances of this crash on 6.1 kernel during stability testing (Involves IO, reboots, device suspend/resume, and few more).
With the V1 patch, we didn't observe the issue for at least 48hrs+ of stability testing.

--
Damien Le Moal
Western Digital Research