linux-ext4 - Re: kernel panics when hot removing U.2 nvme disk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200924051230.GA16433@infradead.org>
Date:   Thu, 24 Sep 2020 06:12:30 +0100
From:   Christoph Hellwig <hch@...radead.org>
To:     Keith Busch <kbusch@...nel.org>
Cc:     Meng Wang <meng@...atainc.com>, linux-ext4@...r.kernel.org,
        "linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>
Subject: Re: kernel panics when hot removing U.2 nvme disk

On Fri, Sep 18, 2020 at 06:44:01PM -0700, Keith Busch wrote:
> On Fri, Sep 18, 2020 at 11:47:27PM +0000, Meng Wang wrote:
> > Hi,
> > We found kernel panics today when doing test on hot remove U.2 nvme
> > disk. After hot remove the nvme disk (formatted as ext4), the system
> > freezes and all services stuck. Lot of kernel message flushed the
> > syslog, including the CPU soft lockup, ext4 NULL point er dereferece
> > and ib nic transmission timeout. The kernel panics and configuration
> > are shown below. The used kernel is 5.4.0-050400-generic and OS is
> > Ubuntu 16.04. Not sure whether it's a known bug or configuration
> > error. Any advise are welcome.
> 
> [cc'ing ext4 mailing list]
> 
> The NULL dereference occured before the soft lockup, so I'm guessing the
> Oops'ed process is holding the same lock the removal task wants.
> 
> Your kernel is a bit older, so it may be worth verifying if your
> observation still occurs on the current stable or current mainline, but
> the ext4 developers may have a better idea as this doesn't at least
> initially appear specific to nvme.

The problem is the crazy __invalidate_device stuff that calls into
file system eviction from all kinds of super critical block paths.
While I haven't debugged the root cause this kind of thing just causes
problems without really helping anyone.  I have a half-finished series
that kills this crap and instead allows the file system (or other
block device user) to pass shutdown and resize callbacks when the
exclusively open a block device.  That way the file system driver
can just mark the file system shutdown to prevent any further damage
without all this mess.