linux-kernel - Re: [PATCH] io_uring: One wqe per wq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a722515b-1311-2abc-c5ab-420609e7131b@gmail.com>
Date:   Mon, 13 Mar 2023 03:56:41 +0000
From:   Pavel Begunkov <asml.silence@...il.com>
To:     Jens Axboe <axboe@...nel.dk>, Breno Leitao <leitao@...ian.org>,
        io-uring@...r.kernel.org
Cc:     leit@...com, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] io_uring: One wqe per wq

On 3/11/23 22:13, Jens Axboe wrote:
> On 3/11/23 1:56 PM, Pavel Begunkov wrote:
>> On 3/10/23 20:38, Jens Axboe wrote:
>>> On 3/10/23 1:11 PM, Breno Leitao wrote:
>>>> Right now io_wq allocates one io_wqe per NUMA node.  As io_wq is now
>>>> bound to a task, the task basically uses only the NUMA local io_wqe, and
>>>> almost never changes NUMA nodes, thus, the other wqes are mostly
>>>> unused.
>>>
>>> What if the task gets migrated to a different node? Unless the task
>>> is pinned to a node/cpumask that is local to that node, it will move
>>> around freely.
>>
>> In which case we're screwed anyway and not only for the slow io-wq
>> path but also with the hot path as rings and all io_uring ctx and
>> requests won't be migrated locally.
> 
> Oh agree, not saying it's ideal, but it can happen.
> 
> What if you deliberately use io-wq to offload work and you set it
> to another mask? That one I supposed we could handle by allocating
> based on the set mask. Two nodes might be more difficult...
> 
> For most things this won't really matter as io-wq is a slow path
> for that, but there might very well be cases that deliberately
> offload.

It's not created for that, there is no fine control by the user.
If the user set affinity solely to another node, then it will
be quite bad for perf, if the mask covers multiple nodes, it'll
go to the current node. Do you have plans on io-wq across
numa nodes?


>> It's also curious whether io-wq workers will get migrated
>> automatically as they are a part of the thread group.
> 
> They certainly will, unless affinitized otherwise.

-- 
Pavel Begunkov