linux-kernel - Re: [PATCH v7 05/10] scsi: Use block layer helpers to constrain queue affinity

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c06d0ef2-6bd2-4f5d-895f-0255415b2a24@suse.de>
Date: Fri, 4 Jul 2025 12:28:49 +0200
From: Hannes Reinecke <hare@...e.de>
To: Daniel Wagner <dwagner@...e.de>
Cc: Daniel Wagner <wagi@...nel.org>, Jens Axboe <axboe@...nel.dk>,
 Keith Busch <kbusch@...nel.org>, Christoph Hellwig <hch@....de>,
 Sagi Grimberg <sagi@...mberg.me>, "Michael S. Tsirkin" <mst@...hat.com>,
 Aaron Tomlin <atomlin@...mlin.com>,
 "Martin K. Petersen" <martin.petersen@...cle.com>,
 Thomas Gleixner <tglx@...utronix.de>, Costa Shulyupin
 <costa.shul@...hat.com>, Juri Lelli <juri.lelli@...hat.com>,
 Valentin Schneider <vschneid@...hat.com>, Waiman Long <llong@...hat.com>,
 Ming Lei <ming.lei@...hat.com>, Frederic Weisbecker <frederic@...nel.org>,
 Mel Gorman <mgorman@...e.de>,
 Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
 linux-kernel@...r.kernel.org, linux-block@...r.kernel.org,
 linux-nvme@...ts.infradead.org, megaraidlinux.pdl@...adcom.com,
 linux-scsi@...r.kernel.org, storagedev@...rochip.com,
 virtualization@...ts.linux.dev, GR-QLogic-Storage-Upstream@...vell.com
Subject: Re: [PATCH v7 05/10] scsi: Use block layer helpers to constrain queue
 affinity

On 7/4/25 11:37, Daniel Wagner wrote:
> On Thu, Jul 03, 2025 at 08:43:01AM +0200, Hannes Reinecke wrote:
>> All of these drivers are not aware of CPU hotplug, and as such
>> will not be notified when the number of CPUs changes.
> 
> Ah, this explains this part.
> 
>> But you use 'blk_mq_online_queue_affinity()' for all of these
>> drivers.
> 
> All these drivers are also using blk_mq_num_online_queue. When I only
> used cpu_possible_mask the resulting mapping was not usable.
> 
Yeah, I'd imagine so. Quite some drivers have 'interesting' ideas how
the firmware interface should look like.

But it also means that there is a very high likelyhood that these
drivers become inoperable under CPU hotplug.
Is there a way of disabling CPU hotplug when these drivers are in use?

>> Wouldn't 'blk_mq_possible_queue_affinit()' a better choice here
>> to insulate against CPU hotplug effects?
> 
> With this mask the queues will be distributed to all possible CPUs and
> some of the hardware queues could be assigned to offline CPUs. I think
> this would work but the question is, is this okay to leave some of the
> perfomance on the road?
> 
It really shouldn't be an issue when the cpus are distributed 
'correctly' :-)
We have several possibilities:
-> #hwq > num_possible_cpus: easy, 1:1 mapping, no problem
-> num_online_cpu < #hwq < num_possible_cpus: Not as easy, but if we
    ensure that each online cpu is mapped to a different hwq we don't
    have a performance impact.
-> #hwq < num_online_cpu: If we ensure that a) the number of online cpus
    per hwq is (roughly) identical we won't have a performance impact.
    As a bonus we should strive to have the number of offline cpus
    distributed equally on each hwq.

Of course, that doesn't take into accound NUMA locality; with NUMA 
locality you would need to ensure to have at least one CPU per NUMA node
mapped to each hwq. Which actually would impose a lower limit on the
number (and granularity!) of hwqs (namely the number of NUMA nodes), but 
that's fair, I guess.

But this really can be delegated to later patches; initially we really
should identify which drivers might have issues with CPU hotplug,
and at the very least issue a warning for these drivers.

> I am not against this, just saying it would change the existing
> behavior.
> 

Oh, sure. No-one (except lpfc on Power) is testing CPU hotplug actively.

>> Also some drivers which are using irq affinity (eg aacraid, lpfc) are
>> missing from these conversions. Why?
> 
> I was not aware of aacraid. I started to work on lpfc and well let's put
> it this way, it's complicated. lpfc needs a lot of work to make it
> isolcpus aware.

Yeah, I know. Sorry ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@...e.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich