netdev - Re: [RFC PATCH net-next v6 14/15] net: add devmem TCP documentation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Thu, 7 Mar 2024 17:52:51 -0800
From: Jakub Kicinski <kuba@...nel.org>
To: Mina Almasry <almasrymina@...gle.com>
Cc: netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
 linux-doc@...r.kernel.org, linux-alpha@...r.kernel.org,
 linux-mips@...r.kernel.org, linux-parisc@...r.kernel.org,
 sparclinux@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 linux-arch@...r.kernel.org, bpf@...r.kernel.org,
 linux-kselftest@...r.kernel.org, linux-media@...r.kernel.org,
 dri-devel@...ts.freedesktop.org, "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>,
 Jonathan Corbet <corbet@....net>, Richard Henderson
 <richard.henderson@...aro.org>, Ivan Kokshaysky <ink@...assic.park.msu.ru>,
 Matt Turner <mattst88@...il.com>, Thomas Bogendoerfer
 <tsbogend@...ha.franken.de>, "James E.J. Bottomley"
 <James.Bottomley@...senPartnership.com>, Helge Deller <deller@....de>,
 Andreas Larsson <andreas@...sler.com>, Jesper Dangaard Brouer
 <hawk@...nel.org>, Ilias Apalodimas <ilias.apalodimas@...aro.org>, Steven
 Rostedt <rostedt@...dmis.org>, Masami Hiramatsu <mhiramat@...nel.org>,
 Mathieu Desnoyers <mathieu.desnoyers@...icios.com>, Arnd Bergmann
 <arnd@...db.de>, Alexei Starovoitov <ast@...nel.org>, Daniel Borkmann
 <daniel@...earbox.net>, Andrii Nakryiko <andrii@...nel.org>, Martin KaFai
 Lau <martin.lau@...ux.dev>, Eduard Zingerman <eddyz87@...il.com>, Song Liu
 <song@...nel.org>, Yonghong Song <yonghong.song@...ux.dev>, John Fastabend
 <john.fastabend@...il.com>, KP Singh <kpsingh@...nel.org>, Stanislav
 Fomichev <sdf@...gle.com>, Hao Luo <haoluo@...gle.com>, Jiri Olsa
 <jolsa@...nel.org>, David Ahern <dsahern@...nel.org>, Willem de Bruijn
 <willemdebruijn.kernel@...il.com>, Shuah Khan <shuah@...nel.org>, Sumit
 Semwal <sumit.semwal@...aro.org>, "Christian König"
 <christian.koenig@....com>, Pavel Begunkov <asml.silence@...il.com>, David
 Wei <dw@...idwei.uk>, Jason Gunthorpe <jgg@...pe.ca>, Yunsheng Lin
 <linyunsheng@...wei.com>, Shailend Chand <shailend@...gle.com>, Harshitha
 Ramamurthy <hramamurthy@...gle.com>, Shakeel Butt <shakeelb@...gle.com>,
 Jeroen de Borst <jeroendb@...gle.com>, Praveen Kaligineedi
 <pkaligineedi@...gle.com>
Subject: Re: [RFC PATCH net-next v6 14/15] net: add devmem TCP documentation

On Mon,  4 Mar 2024 18:01:49 -0800 Mina Almasry wrote:
> +Intro
> +=====
> +
> +Device memory TCP (devmem TCP) enables receiving data directly into device
> +memory (dmabuf). The feature is currently implemented for TCP sockets.
> +
> +
> +Opportunity
> +-----------
> +
> +A large amount of data transfers have device memory as the source and/or

s/amount/number/

> +destination. Accelerators drastically increased the volume of such transfers.

s/volume/prevalence/

> +Some examples include:
> +
> +- Distributed training, where ML accelerators, such as GPUs on different hosts,
> +  exchange data among them.

s/among them//

> +- Distributed raw block storage applications transfer large amounts of data with
> +  remote SSDs, much of this data does not require host processing.
> +
> +Today, the majority of the Device-to-Device data transfers the network are

"Today" won't age well.

> +implemented as the following low level operations: Device-to-Host copy,
> +Host-to-Host network transfer, and Host-to-Device copy.
> +
> +The implementation is suboptimal, especially for bulk data transfers, and can

/The implementation/The flow involving host copies/

> +put significant strains on system resources such as host memory bandwidth and
> +PCIe bandwidth.
> +
> +Devmem TCP optimizes this use case by implementing socket APIs that enable
> +the user to receive incoming network packets directly into device memory.

> +More Info
> +---------
> +
> +  slides, video
> +    https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html
> +
> +  patchset
> +    [RFC PATCH v3 00/12] Device Memory TCP
> +    https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@google.com/T/

Won't age well? :)

> +Interface
> +=========
> +
> +Example
> +-------
> +
> +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up
> +the RX path of this API.
> +
> +NIC Setup
> +---------
> +
> +Header split, flow steering, & RSS are required features for devmem TCP.
> +
> +Header split is used to split incoming packets into a header buffer in host
> +memory, and a payload buffer in device memory.
> +
> +Flow steering & RSS are used to ensure that only flows targeting devmem land on
> +RX queue bound to devmem.
> +
> +Enable header split & flow steering:
> +
> +::

You can put the :: at the end of the text, IIRC, like this:

Enable header split & flow steering::

> +
> +	# enable header split (assuming priv-flag)
> +	ethtool --set-priv-flags eth1 enable-header-split on

Olek added the "set" in commit 50d73710715d ("ethtool: add SET for
TCP_DATA_SPLIT ringparam"), no need for the priv flag any more.