Message-ID: <afcb3c40-0148-46ef-b2be-fa4adc57b88a@davidwei.uk>
Date: Sun, 22 Oct 2023 20:35:44 -0700
From: David Wei <dw@...idwei.uk>
To: Gal Pressman <gal@...dia.com>, Jens Axboe <axboe@...nel.dk>,
 Pavel Begunkov <asml.silence@...il.com>
Cc: io-uring@...r.kernel.org, netdev@...r.kernel.org,
 Mina Almasry <almasrymina@...gle.com>, Jakub Kicinski <kuba@...nel.org>
Subject: Re: [RFC RESEND 00/11] Zero copy network RX using io_uring

On 2023-10-22 12:06, Gal Pressman wrote:
> On 26/08/2023 4:19, David Wei wrote:
>> From: David Wei <davidhwei@...a.com>
>>
>> This patchset is a proposal that adds zero copy network RX to io_uring.
>> With it, userspace can register a region of host memory for receiving
>> data directly from a NIC using DMA, without needing a kernel to user
>> copy.
>>
>> Software support is added to the Broadcom BNXT driver. Hardware support
>> for receive flow steering and header splitting is required.
>>
>> On the userspace side, a sample server is added in this branch of
>> liburing:
>> https://github.com/spikeh/liburing/tree/zcrx2
>>
>> Build liburing as normal, and run examples/zcrx. Then, set flow steering
>> rules using ethtool. A sample shell script is included in
>> examples/zcrx_flow.sh, but you need to change the source IP. Finally,
>> connect a client using e.g. netcat and send data.
>>
>> This patchset + userspace code was tested on an Intel Xeon Platinum
>> 8321HC CPU and Broadcom BCM57504 NIC.
>>
>> Early benchmarks using this prototype, with iperf3 as a load generator,
>> showed a ~50% reduction in overall system memory bandwidth as measured
>> using perf counters. Note that DDIO must be disabled on Intel systems.
>>
>> Mina et al. from Google and Kuba are collaborating on a similar proposal
>> to ZC from NIC to devmem. There are many shared functionality in netdev
>> that we can collaborate on e.g.:
>> * Page pool memory provider backend and resource registration
>> * Page pool refcounted iov/buf representation and lifecycle
>> * Setting receive flow steering
>>
>> As mentioned earlier, this is an early prototype. It is brittle, some
>> functionality is missing and there's little optimisation. We're looking
>> for feedback on the overall approach and points of collaboration in
>> netdev.
>> * No copy fallback, if payload ends up in linear part of skb then the
>>   code will not work
>> * No way to pin an RX queue to a specific CPU
>> * Only one ifq, one pool region, on RX queue...
>>
>> This patchset is based on the work by Jonathan Lemon
>> <jonathan.lemon@...il.com>:
>> https://lore.kernel.org/io-uring/20221108050521.3198458-1-jonathan.lemon@gmail.com/
> 
> Hello David,
> 
> This work looks interesting, is there anywhere I can read about it some
> more? Maybe it was presented (and hopefully recorded) in a recent
> conference?
> Maybe something geared towards adding more drivers support?
> 

Hi Gal,

Thank you for your interest in our work! We will be publishing a paper
and presenting this work at NetDev conference on 1 Nov.

Support for more drivers (e.g. mlx5) is definitely on our radar. We are
collaborating with Mina and others from Google who are working on a
similar proposal but targetting NIC -> ZC RX into GPU memory. We both
require shared bits of infra e.g. page pool memory providers that will
replace the use of a one-off data_pool in this patchset. This would
minimise driver changes needed to support this feature.

> I took a brief look at the bnxt patch and saw you converted the page
> pool allocation to data pool allocation, I assume this is done for data
> pages only, right? Headers are still allocated on page pool pages?
> 
> Thanks

Yes, that's right.

David

Powered by blists - more mailing lists