linux-kernel - RE: Random high CPU utilization in blk-mq with the none scheduler

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BYAPR21MB1270B53CFDDFD52AD942B3AEBF729@BYAPR21MB1270.namprd21.prod.outlook.com>
Date:   Sat, 11 Dec 2021 07:09:45 +0000
From:   Dexuan Cui <decui@...rosoft.com>
To:     Jens Axboe <axboe@...nel.dk>,
        "'ming.lei@...hat.com'" <ming.lei@...hat.com>,
        'Christoph Hellwig' <hch@....de>,
        "'linux-block@...r.kernel.org'" <linux-block@...r.kernel.org>
CC:     Long Li <longli@...rosoft.com>,
        "Michael Kelley (LINUX)" <mikelley@...rosoft.com>,
        "'linux-kernel@...r.kernel.org'" <linux-kernel@...r.kernel.org>
Subject: RE: Random high CPU utilization in blk-mq with the none scheduler

> From: Dexuan Cui
> Sent: Friday, December 10, 2021 7:45 PM
> 
> > From: Jens Axboe <axboe@...nel.dk>
> >
> > Just out of curiosity, can you do:
> >
> > # perf record -a -g -- sleep 3
> >
> > when you see the excessive CPU usage, then attach the output of
> >
> > # perf report -g
> >
> > to a reply?

I realized you only asked for the output of "pref report -g", which
is much smaller. Please see the attachment for it. 
try_to_grab_pending() is the hottest function, e.g. see line 2479.

It's generated using v5.16-rc4 with the 3 commits reverted:
b22809092c70 ("block: replace always false argument with 'false'")
ff1552232b36 ("blk-mq: don't issue request directly in case that current is to be blocked")
dc5fc361d891 ("block: attempt direct issue of plug list")

> > How confident are you in your bisect result?
> >
> > Jens Axboe

I did some tests again and I'm pretty confident that the commit
b22809092c70 resolves the excessive CPU usage.

Thanks,
Dexuan

Download attachment "perf.data.output.log.bz2" of type "application/octet-stream" (129840 bytes)