linux-kernel - Re: 4.19 LTS high /proc/diskstats io

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <8b876ed9753478764bf5ea74d00b3d54c381b9f2.camel@nokia.com>
Date:   Wed, 25 Mar 2020 11:22:26 +0000
From:   "Rantala, Tommi T. (Nokia - FI/Espoo)" <tommi.t.rantala@...ia.com>
To:     "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>
CC:     "snitzer@...hat.com" <snitzer@...hat.com>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>,
        "axboe@...nel.dk" <axboe@...nel.dk>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "osandov@...com" <osandov@...com>,
        "linux-block@...r.kernel.org" <linux-block@...r.kernel.org>,
        "mpatocka@...hat.com" <mpatocka@...hat.com>
Subject: Re: 4.19 LTS high /proc/diskstats io_ticks

On Wed, 2020-03-25 at 11:07 +0100, Greg KH wrote:
> On Wed, Mar 25, 2020 at 10:02:41AM +0000, Rantala, Tommi T. (Nokia -
> FI/Espoo) wrote:
> > 
> > Other people are apparently seeing this too with 4.19:
> > https://kudzia.eu/b/2019/09/iostat-x-1-reporting-100-utilization-of-nearly-idle-nvme-drives/
> > 
> > 
> > I also see this only in 4.19.y and bisected to this (based on the Fixes
> > tag, this should have been taken to 4.14 too...):
> > 
> > commit 6131837b1de66116459ef4413e26fdbc70d066dc
> > Author: Omar Sandoval <osandov@...com>
> > Date:   Thu Apr 26 00:21:58 2018 -0700
> > 
> >   blk-mq: count allocated but not started requests in iostats inflight
> > 
> >   In the legacy block case, we increment the counter right after we
> >   allocate the request, not when the driver handles it. In both the
> > legacy
> >   and blk-mq cases, part_inc_in_flight() is called from
> >   blk_account_io_start() right after we've allocated the request. blk-mq
> >   only considers requests started requests as inflight, but this is
> >   inconsistent with the legacy definition and the intention in the code.
> >   This removes the started condition and instead counts all allocated
> >   requests.
> > 
> >   Fixes: f299b7c7a9de ("blk-mq: provide internal in-flight variant")
> >   Signed-off-by: Omar Sandoval <osandov@...com>
> >   Signed-off-by: Jens Axboe <axboe@...nel.dk>
> > 
> > 
> > 
> > If I get it right, when the disk is idle, and some request is allocated,
> > part_round_stats() with this commit will now add all ticks between
> > previous I/O and current time (now - part->stamp) to io_ticks.
> > 
> > Before the commit, part_round_stats() would only update part->stamp when
> > called after request allocation.
> 
> So this is a "false" reporting?  there's really no load?

Correct.

> > Any thoughts how to best fix this in 4.19?
> > I see the io_ticks accounting has been reworked in 5.0, do we need to
> > backport those to 4.19, or any ill effects if this commit is reverted in
> > 4.19?
> 
> Do you see this issue in 5.4?  What's keeping you from moving to 5.4.y?

Yes it's fixed in 5.0 and later.
Fixing it in 4.19.x would benefit debian stable users too. :-)
BTW is the EOL for 4.19 still end of 2020?

> And if this isn't a real issue, is that a problem too?
> 
> As you can test this, if you have a set of patches backported that could
> resolve it, can you send them to us?

Based on quick looks it's solved by this, I'll need to check if it's enough
to fix it:

commit 5b18b5a737600fd20ba2045f320d5926ebbf341a
Author: Mikulas Patocka <mpatocka@...hat.com>
Date:   Thu Dec 6 11:41:19 2018 -0500

    block: delete part_round_stats and switch to less precise counting




> 
> thanks,
> 
> greg k-h