Date: Tue, 11 Jul 2023 20:56:56 -0300
From: Jason Gunthorpe <jgg@...pe.ca>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Christoph Hellwig <hch@....de>, Mina Almasry <almasrymina@...gle.com>,
	John Hubbard <jhubbard@...dia.com>,
	Dan Williams <dan.j.williams@...el.com>,
	David Ahern <dsahern@...nel.org>,
	Jesper Dangaard Brouer <jbrouer@...hat.com>, brouer@...hat.com,
	Alexander Duyck <alexander.duyck@...il.com>,
	Yunsheng Lin <linyunsheng@...wei.com>, davem@...emloft.net,
	pabeni@...hat.com, netdev@...r.kernel.org,
	linux-kernel@...r.kernel.org, Lorenzo Bianconi <lorenzo@...nel.org>,
	Yisen Zhuang <yisen.zhuang@...wei.com>,
	Salil Mehta <salil.mehta@...wei.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Sunil Goutham <sgoutham@...vell.com>,
	Geetha sowjanya <gakula@...vell.com>,
	Subbaraya Sundeep <sbhatta@...vell.com>,
	hariprasad <hkelam@...vell.com>, Saeed Mahameed <saeedm@...dia.com>,
	Leon Romanovsky <leon@...nel.org>, Felix Fietkau <nbd@....name>,
	Ryder Lee <ryder.lee@...iatek.com>,
	Shayne Chen <shayne.chen@...iatek.com>,
	Sean Wang <sean.wang@...iatek.com>, Kalle Valo <kvalo@...nel.org>,
	Matthias Brugger <matthias.bgg@...il.com>,
	AngeloGioacchino Del Regno <angelogioacchino.delregno@...labora.com>,
	Jesper Dangaard Brouer <hawk@...nel.org>,
	Ilias Apalodimas <ilias.apalodimas@...aro.org>,
	linux-rdma@...r.kernel.org, linux-wireless@...r.kernel.org,
	linux-arm-kernel@...ts.infradead.org,
	linux-mediatek@...ts.infradead.org,
	Jonathan Lemon <jonathan.lemon@...il.com>
Subject: Re: Memory providers multiplexing (Was: [PATCH net-next v4 4/5]
 page_pool: remove PP_FLAG_PAGE_FRAG flag)

On Tue, Jul 11, 2023 at 01:34:20PM -0700, Jakub Kicinski wrote:

> > Yep. At the high end open standards based ethernet has also notably
> > "failed" as well. Every switch vendor now offers their own proprietary
> > ecosystem on a whole bunch of different axis. They all present
> > "ethernet" toward the host but the host often needs to work in a
> > special way to really take full advantage of the proprietary fabric
> > behaviors.
> 
> I'm not familiar with "high end open standards based on ethernet", would
> those be some RDMA / storage things? For TCP/IP networks pretty much
> the only things that matter in a switch are bandwidth, size of buffers,
> power... Implementation stuff.

I would say when you are getting into ethernet deployments with 25 or
51 Tbps switches directly connected to hosts running at >100G you are
getting into the high end side of things.

These are very expensive networks. They run complex congestion
provoking workloads. They have sophisticated multi-pathing. They often
use use a non-blocking topology. Congestion management is important.

Making this work with good utilization, and low tail latency is a
really hard problem. Many of the solutions come with switch features
supporting it.

You'd proably say these are not TCP focused networks, even though they
are based on ethernet and IP.

So I think of them as high end "standards based" ethernet and IP
looking networks that have proprietary elements mixed in throughout.

Right now there is a bit of a press war between vendors on 'ethernet
for AI'. Both Broadcom and NVIDIA are taking techonlogies that were
originally built for TCP ethernet networks and remixing/speeding them
up to run roce workloads effectively. There is alot more information
available now without NDA that shows some detail on this space.

AWS's SRD multipathing, Broadcom "AI Ethernet" and NVIDIA's Spectrum-X
spring to mind as topical to what these sorts of ethernet networks
are.

> A lot of "standardization" efforts are just attempts to prove to 
> a buyers that an ecosystem exists.

Heh, that was probably more true years ago. These days it seems like
some standardization is also being done so the large hyperscalers can
improve their Approved Vendors List.

I suppose as long as the result is something we can implement openly
in Linux the motivation for standardization is less important.

Jason

Powered by blists - more mailing lists