netdev - Re: [PATCH net-next] xsk: introduce xsk_dma

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d18eea7a-a71c-8de0-bde3-7ab000a77539@intel.com>
Date:   Wed, 19 Apr 2023 15:14:48 +0200
From:   Alexander Lobakin <aleksander.lobakin@...el.com>
To:     Christoph Hellwig <hch@...radead.org>,
        Jakub Kicinski <kuba@...nel.org>
CC:     Xuan Zhuo <xuanzhuo@...ux.alibaba.com>, <netdev@...r.kernel.org>,
        Björn Töpel <bjorn@...nel.org>,
        Magnus Karlsson <magnus.karlsson@...el.com>,
        Maciej Fijalkowski <maciej.fijalkowski@...el.com>,
        Jonathan Lemon <jonathan.lemon@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Paolo Abeni <pabeni@...hat.com>,
        Alexei Starovoitov <ast@...nel.org>,
        "Daniel Borkmann" <daniel@...earbox.net>,
        Jesper Dangaard Brouer <hawk@...nel.org>,
        John Fastabend <john.fastabend@...il.com>,
        <bpf@...r.kernel.org>, <virtualization@...ts.linux-foundation.org>,
        "Michael S. Tsirkin" <mst@...hat.com>,
        Guenter Roeck <linux@...ck-us.net>,
        Gerd Hoffmann <kraxel@...hat.com>,
        Jason Wang <jasowang@...hat.com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Jens Axboe <axboe@...nel.dk>,
        Linus Torvalds <torvalds@...ux-foundation.org>
Subject: Re: [PATCH net-next] xsk: introduce xsk_dma_ops

From: Christoph Hellwig <hch@...radead.org>
Date: Tue, 18 Apr 2023 22:16:53 -0700

> On Mon, Apr 17, 2023 at 11:19:47PM -0700, Jakub Kicinski wrote:
>>> You can't just do dma mapping outside the driver, because there are
>>> drivers that do not require DMA mapping at all.  virtio is an example,
>>> but all the classic s390 drivers and some other odd virtualization
>>> ones are others.
>>
>> What bus are the classic s390 on (in terms of the device model)?
> 
> I think most of them are based on struct ccw_device, but I'll let the
> s390 maintainers fill in.
> 
> Another interesting case that isn't really relevant for your networking
> guys, but that caused as problems is RDMA.  For hardware RDMA devices
> it wants the ULPs to DMA map, but it turns out we have various software
> drivers that map to network drivers that do their own DMA mapping
> at a much lower layer and after potentially splitting packets or
> even mangling them.
> 
>>
>>>> I don't think it's reasonable to be bubbling up custom per-subsystem
>>>> DMA ops into all of them for the sake of virtio.  
>>>
>>> dma addresses and thus dma mappings are completely driver specific.
>>> Upper layers have no business looking at them.

Here it's not an "upper layer". XSk core doesn't look at them or pass
them between several drivers. It maps DMA solely via the struct device
passed from the driver and then just gets-sets addresses for this driver
only. Just like Page Pool does for regular Rx buffers. This got moved to
the XSk core to not repeat the same code pattern in each driver.

>>
>> Damn, that's unfortunate. Thinking aloud -- that means that if we want 
>> to continue to pull memory management out of networking drivers to
>> improve it for all, cross-optimize with the rest of the stack and
>> allow various upcoming forms of zero copy -- then we need to add an
>> equivalent of dma_ops and DMA API locally in networking?

Managing DMA addresses is totally fine as long as you don't try to pass
mapped addresses between different drivers :D Page Pool already does
that and I don't see a problem in that in general.

> 
> Can you explain what the actual use case is?
> 
>>>From the original patchset I suspect it is dma mapping something very
> long term and then maybe doing syncs on it as needed?

As I mentioned, XSk provides some handy wrappers to map DMA for drivers.
Previously, XSk was supported by real hardware drivers only, but here
the developer tries to add support to virtio-net. I suspect he needs to
use DMA mapping functions different from which the regular driver use.
So this is far from dma_map_ops, the author picked wrong name :D
And correct, for XSk we map one big piece of memory only once and then
reuse it for buffers, no inflight map/unmap on hotpath (only syncs when
needed). So this mapping is longterm and is stored in XSk core structure
assigned to the driver which this mapping was done for.
I think Jakub thinks of something similar, but for the "regular" Rx/Tx,
not only XDP sockets :)

Thanks,
Olek