linux-kernel - Re: Random high CPU utilization in blk-mq with the none scheduler

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c5c8f95e-f430-6655-bab5-d2a2948ab81d@kernel.dk>
Date:   Fri, 10 Dec 2021 20:15:39 -0700
From:   Jens Axboe <axboe@...nel.dk>
To:     Dexuan Cui <decui@...rosoft.com>,
        "'ming.lei@...hat.com'" <ming.lei@...hat.com>,
        'Christoph Hellwig' <hch@....de>,
        "'linux-block@...r.kernel.org'" <linux-block@...r.kernel.org>
Cc:     Long Li <longli@...rosoft.com>,
        "Michael Kelley (LINUX)" <mikelley@...rosoft.com>,
        "'linux-kernel@...r.kernel.org'" <linux-kernel@...r.kernel.org>
Subject: Re: Random high CPU utilization in blk-mq with the none scheduler

On 12/10/21 8:10 PM, Dexuan Cui wrote:
>> From: Jens Axboe <axboe@...nel.dk>
>> Sent: Friday, December 10, 2021 6:05 PM
>> ...
>> It's more likely the real fix is avoiding the repeated plug list scan,
>> which I guess makes sense. That is this commit:
>>
>> commit d38a9c04c0d5637a828269dccb9703d42d40d42b
>> Author: Jens Axboe <axboe@...nel.dk>
>> Date:   Thu Oct 14 07:24:07 2021 -0600
>>
>>     block: only check previous entry for plug merge attempt
>>
>> If that's the case, try 5.15.x again and do:
>>
>> echo 2 > /sys/block/<dev>/queue/nomerges
>>
>> for each drive you are using in the IO test, and see if that gets
>> rid of the excess CPU usage.
>>
>> --
>> Jens Axboe
> 
> Thanks for the reply! Unluckily this does not work.
> 
> I tried the below command:
> 
> for i in `ls /sys/block/*/queue/nomerges`; do echo 2 > $i; done
> 
> and verified that the "nomerges" are changed to "2", but the
> excess CPU usage can still reproduce easily.

Just out of curiosity, can you do:

# perf record -a -g -- sleep 3

when you see the excessive CPU usage, then attach the output of

# perf report -g

to a reply?

How confident are you in your bisect result?

-- 
Jens Axboe