linux-kernel - RE: Random high CPU utilization in blk-mq with the none scheduler

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <BYAPR21MB1270B5DAD526C42C070ECB9EBF729@BYAPR21MB1270.namprd21.prod.outlook.com>
Date:   Sat, 11 Dec 2021 03:10:43 +0000
From:   Dexuan Cui <decui@...rosoft.com>
To:     Jens Axboe <axboe@...nel.dk>,
        "'ming.lei@...hat.com'" <ming.lei@...hat.com>,
        'Christoph Hellwig' <hch@....de>,
        "'linux-block@...r.kernel.org'" <linux-block@...r.kernel.org>
CC:     Long Li <longli@...rosoft.com>,
        "Michael Kelley (LINUX)" <mikelley@...rosoft.com>,
        "'linux-kernel@...r.kernel.org'" <linux-kernel@...r.kernel.org>
Subject: RE: Random high CPU utilization in blk-mq with the none scheduler

> From: Jens Axboe <axboe@...nel.dk>
> Sent: Friday, December 10, 2021 6:05 PM
> ...
> It's more likely the real fix is avoiding the repeated plug list scan,
> which I guess makes sense. That is this commit:
> 
> commit d38a9c04c0d5637a828269dccb9703d42d40d42b
> Author: Jens Axboe <axboe@...nel.dk>
> Date:   Thu Oct 14 07:24:07 2021 -0600
> 
>     block: only check previous entry for plug merge attempt
> 
> If that's the case, try 5.15.x again and do:
> 
> echo 2 > /sys/block/<dev>/queue/nomerges
> 
> for each drive you are using in the IO test, and see if that gets
> rid of the excess CPU usage.
> 
> --
> Jens Axboe

Thanks for the reply! Unluckily this does not work.

I tried the below command:

for i in `ls /sys/block/*/queue/nomerges`; do echo 2 > $i; done

and verified that the "nomerges" are changed to "2", but the
excess CPU usage can still reproduce easily.

(BTW, I got my 5.15.7 kernel from here:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.15.7/)

root@...alhost:~# uname -r
5.15.7-051507-generic
root@...alhost:~# grep  . /sys/block/*/queue/nomerges
/sys/block/dm-0/queue/nomerges:2
/sys/block/dm-1/queue/nomerges:2
/sys/block/dm-2/queue/nomerges:2
/sys/block/dm-3/queue/nomerges:2
/sys/block/loop0/queue/nomerges:2
/sys/block/loop1/queue/nomerges:2
/sys/block/loop2/queue/nomerges:2
/sys/block/loop3/queue/nomerges:2
/sys/block/loop4/queue/nomerges:2
/sys/block/loop5/queue/nomerges:2
/sys/block/loop6/queue/nomerges:2
/sys/block/loop7/queue/nomerges:2
/sys/block/sda/queue/nomerges:2
/sys/block/sdaa/queue/nomerges:2
/sys/block/sdab/queue/nomerges:2
/sys/block/sdac/queue/nomerges:2
/sys/block/sdad/queue/nomerges:2
/sys/block/sdae/queue/nomerges:2
/sys/block/sdaf/queue/nomerges:2
/sys/block/sdag/queue/nomerges:2
/sys/block/sdah/queue/nomerges:2
/sys/block/sdai/queue/nomerges:2
/sys/block/sdaj/queue/nomerges:2
/sys/block/sdak/queue/nomerges:2
/sys/block/sdal/queue/nomerges:2
/sys/block/sdam/queue/nomerges:2
/sys/block/sdan/queue/nomerges:2
/sys/block/sdao/queue/nomerges:2
/sys/block/sdap/queue/nomerges:2
/sys/block/sdaq/queue/nomerges:2
/sys/block/sdar/queue/nomerges:2
/sys/block/sdas/queue/nomerges:2
/sys/block/sdat/queue/nomerges:2
/sys/block/sdau/queue/nomerges:2
/sys/block/sdav/queue/nomerges:2
/sys/block/sdaw/queue/nomerges:2
/sys/block/sdax/queue/nomerges:2
/sys/block/sday/queue/nomerges:2
/sys/block/sdaz/queue/nomerges:2
/sys/block/sdb/queue/nomerges:2
/sys/block/sdba/queue/nomerges:2
/sys/block/sdbb/queue/nomerges:2
/sys/block/sdbc/queue/nomerges:2
/sys/block/sdbd/queue/nomerges:2
/sys/block/sdbe/queue/nomerges:2
/sys/block/sdbf/queue/nomerges:2
/sys/block/sdbg/queue/nomerges:2
/sys/block/sdbh/queue/nomerges:2
/sys/block/sdbi/queue/nomerges:2
/sys/block/sdbj/queue/nomerges:2
/sys/block/sdbk/queue/nomerges:2
/sys/block/sdbl/queue/nomerges:2
/sys/block/sdbm/queue/nomerges:2
/sys/block/sdbn/queue/nomerges:2
/sys/block/sdbo/queue/nomerges:2
/sys/block/sdbp/queue/nomerges:2
/sys/block/sdbq/queue/nomerges:2
/sys/block/sdbr/queue/nomerges:2
/sys/block/sdbs/queue/nomerges:2
/sys/block/sdbt/queue/nomerges:2
/sys/block/sdbu/queue/nomerges:2
/sys/block/sdbv/queue/nomerges:2
/sys/block/sdbw/queue/nomerges:2
/sys/block/sdc/queue/nomerges:2
/sys/block/sdd/queue/nomerges:2
/sys/block/sde/queue/nomerges:2
/sys/block/sdf/queue/nomerges:2
/sys/block/sdg/queue/nomerges:2
/sys/block/sdh/queue/nomerges:2
/sys/block/sdi/queue/nomerges:2
/sys/block/sdj/queue/nomerges:2
/sys/block/sdk/queue/nomerges:2
/sys/block/sdl/queue/nomerges:2
/sys/block/sdm/queue/nomerges:2
/sys/block/sdn/queue/nomerges:2
/sys/block/sdo/queue/nomerges:2
/sys/block/sdp/queue/nomerges:2
/sys/block/sdq/queue/nomerges:2
/sys/block/sdr/queue/nomerges:2
/sys/block/sds/queue/nomerges:2
/sys/block/sdt/queue/nomerges:2
/sys/block/sdu/queue/nomerges:2
/sys/block/sdv/queue/nomerges:2
/sys/block/sdw/queue/nomerges:2
/sys/block/sdx/queue/nomerges:2
/sys/block/sdy/queue/nomerges:2
/sys/block/sdz/queue/nomerges:2