netdev - Re: [RFC RESEND 00/11] Zero copy network RX using io

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1673427d-b449-4f9e-b344-027c0dc2ec9f@nvidia.com>
Date: Sun, 22 Oct 2023 22:06:24 +0300
From: Gal Pressman <gal@...dia.com>
To: David Wei <dw@...idwei.uk>, Jens Axboe <axboe@...nel.dk>,
 Pavel Begunkov <asml.silence@...il.com>
Cc: io-uring@...r.kernel.org, netdev@...r.kernel.org,
 Mina Almasry <almasrymina@...gle.com>, Jakub Kicinski <kuba@...nel.org>
Subject: Re: [RFC RESEND 00/11] Zero copy network RX using io_uring

On 26/08/2023 4:19, David Wei wrote:
> From: David Wei <davidhwei@...a.com>
> 
> This patchset is a proposal that adds zero copy network RX to io_uring.
> With it, userspace can register a region of host memory for receiving
> data directly from a NIC using DMA, without needing a kernel to user
> copy.
> 
> Software support is added to the Broadcom BNXT driver. Hardware support
> for receive flow steering and header splitting is required.
> 
> On the userspace side, a sample server is added in this branch of
> liburing:
> https://github.com/spikeh/liburing/tree/zcrx2
> 
> Build liburing as normal, and run examples/zcrx. Then, set flow steering
> rules using ethtool. A sample shell script is included in
> examples/zcrx_flow.sh, but you need to change the source IP. Finally,
> connect a client using e.g. netcat and send data.
> 
> This patchset + userspace code was tested on an Intel Xeon Platinum
> 8321HC CPU and Broadcom BCM57504 NIC.
> 
> Early benchmarks using this prototype, with iperf3 as a load generator,
> showed a ~50% reduction in overall system memory bandwidth as measured
> using perf counters. Note that DDIO must be disabled on Intel systems.
> 
> Mina et al. from Google and Kuba are collaborating on a similar proposal
> to ZC from NIC to devmem. There are many shared functionality in netdev
> that we can collaborate on e.g.:
> * Page pool memory provider backend and resource registration
> * Page pool refcounted iov/buf representation and lifecycle
> * Setting receive flow steering
> 
> As mentioned earlier, this is an early prototype. It is brittle, some
> functionality is missing and there's little optimisation. We're looking
> for feedback on the overall approach and points of collaboration in
> netdev.
> * No copy fallback, if payload ends up in linear part of skb then the
>   code will not work
> * No way to pin an RX queue to a specific CPU
> * Only one ifq, one pool region, on RX queue...
> 
> This patchset is based on the work by Jonathan Lemon
> <jonathan.lemon@...il.com>:
> https://lore.kernel.org/io-uring/20221108050521.3198458-1-jonathan.lemon@gmail.com/

Hello David,

This work looks interesting, is there anywhere I can read about it some
more? Maybe it was presented (and hopefully recorded) in a recent
conference?
Maybe something geared towards adding more drivers support?

I took a brief look at the bnxt patch and saw you converted the page
pool allocation to data pool allocation, I assume this is done for data
pages only, right? Headers are still allocated on page pool pages?

Thanks