Date:   Mon, 30 Jan 2023 03:09:16 +0000
From:   "Singhai, Anjali" <anjali.singhai@...el.com>
To:     Tom Herbert <tom@...bertland.com>
CC:     Jamal Hadi Salim <hadi@...atatu.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Jamal Hadi Salim <jhs@...atatu.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "kernel@...atatu.com" <kernel@...atatu.com>,
        "Chatterjee, Deb" <deb.chatterjee@...el.com>,
        "Limaye, Namrata" <namrata.limaye@...el.com>,
        "khalidm@...dia.com" <khalidm@...dia.com>,
        "tom@...anda.io" <tom@...anda.io>,
        "pratyush@...anda.io" <pratyush@...anda.io>,
        "jiri@...nulli.us" <jiri@...nulli.us>,
        "xiyou.wangcong@...il.com" <xiyou.wangcong@...il.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "edumazet@...gle.com" <edumazet@...gle.com>,
        "pabeni@...hat.com" <pabeni@...hat.com>,
        "vladbu@...dia.com" <vladbu@...dia.com>,
        "simon.horman@...igine.com" <simon.horman@...igine.com>,
        "stefanc@...vell.com" <stefanc@...vell.com>,
        "seong.kim@....com" <seong.kim@....com>,
        "mattyk@...dia.com" <mattyk@...dia.com>,
        "Daly, Dan" <dan.daly@...el.com>,
        "Fingerhut, John Andy" <john.andy.fingerhut@...el.com>
Subject: RE: [PATCH net-next RFC 00/20] Introducing P4TC

I am agreeing with you Tom. P4tc does not restrict the high level language to be P4, it can be anything as long as it can be compiled to create an IR that can be used to teach/program the SW and the HW, which is what the script-ability of p4tc provides.

Ultimately the devices are evolving as combination of highly efficient Domain specific architecture and the traditional Generic cores, and SW in the kernel has to evolve to program them both in a way that the user can decide whether to run a particular functionality in Domain specific HW or SW that runs on general purpose cores or in some cases the functionality runs in both places and the intelligent (and some-day AI managed Infrastructure controller) entity decides whether the flow should use the HW path or the SW path. There is no other way forward because a SW dataplane can only provide an overflow region for flows and the HW will have to run the most demanding flows, as the Network demand and capacity of the data-center keeps reaching higher and higher levels. From a HW vendor's point of view we have already  entered the 3rd epoch of computer architecture. 

A domain specific architecture still has to be programmable but for a specific domain, linux kernel which has remained fixed function (and fixed protocol) traditionally needs to evolve to support these domain specific architecture that are protocol and dataplane programmable. I think p4tc definitely is the right way forward.
 
There were some arguments made earlier about but the big Datacenters are programming these domain specific architecture from user space already, no doubt but isn't the whole argument for linux kernel is democratizing of the goodness the HW brings to all , the small users and the big ones?

There is also argument that is being made about using ebpf for implementing the SW path, may be I am missing the part as to how do you offload if not to another general purpose core even if it is not as evolved as the current day Xeon's. And we know that even the simplest of the general purpose cores ( example RISC-V) right now cannot sustain the rate at which the network needs to feed the business logic running on the CPUs or GPUs or TPUs in an economically viable solution. All data points to the fact that Network processing running on general purpose cores eats up more than half of the cores and that’s expensive. Because the performance/power unit math when using an IPU/DPU/Smart NIC for network work load is so much better than that of a General purpose core. So I do not see a way forward for epbf to be offloaded on anything but general purpose cores and in the meantime Domain specific programmable ASICs need to be still programmed as they are the right solution for the economy of scale.

Having said that we do have to find a good solution for p4 externs in SW and may be there is room for some helpers ( may be even ebpf) ( as long as you don’t ask me to offload that in HW 😊)

Anjali 

-----Original Message-----
From: Tom Herbert <tom@...bertland.com> 
Sent: Saturday, January 28, 2023 1:18 PM
To: Singhai, Anjali <anjali.singhai@...el.com>
Cc: Jamal Hadi Salim <hadi@...atatu.com>; Jakub Kicinski <kuba@...nel.org>; Jamal Hadi Salim <jhs@...atatu.com>; netdev@...r.kernel.org; kernel@...atatu.com; Chatterjee, Deb <deb.chatterjee@...el.com>; Limaye, Namrata <namrata.limaye@...el.com>; khalidm@...dia.com; tom@...anda.io; pratyush@...anda.io; jiri@...nulli.us; xiyou.wangcong@...il.com; davem@...emloft.net; edumazet@...gle.com; pabeni@...hat.com; vladbu@...dia.com; simon.horman@...igine.com; stefanc@...vell.com; seong.kim@....com; mattyk@...dia.com; Daly, Dan <dan.daly@...el.com>; Fingerhut, John Andy <john.andy.fingerhut@...el.com>
Subject: Re: [PATCH net-next RFC 00/20] Introducing P4TC

On Fri, Jan 27, 2023 at 5:34 PM Singhai, Anjali <anjali.singhai@...el.com> wrote:
>
> P4 is definitely the language of choice for defining a Dataplane in HW for IPUs/DPUs/FNICs and Switches. As a vendor I can definitely say that the smart devices implement a very programmable ASIC as each customer Dataplane defers quite a bit and P4 is the language of choice for specifying the Dataplane definitions. A lot of customer deploy proprietary protocols that run in HW and there is no good way right now in kernel to support these proprietary protcols. If we enable these protocol in the kernel it takes a huge effort and they don’t evolve well.
> Being able to define in P4 and offload into HW using tc mechanism 
> really helps in supporting the customer's Dataplane and protcols 
> without having to wait months and years to get the kernel updated. 
> Here is a link to our IPU offering that is P4 programmable

Anjali,

P4 may be the language of choice for programming HW datapath, however it's not the language of choice for programming SW datapaths-- that's C over XDP/eBPF. And while XDP/eBPF also doesn't depend on kernel updates, it has a major advantage over P4 in that it doesn't require fancy hardware either.

Even at full data center deployment of P4 devices, there will be at least an order of magnitude more deployment of SW programmed datapaths; and unless someone is using P4 hardware, there's zero value in rewriting programs in P4 instead of C. IMO, we will never see networking developers moving to P4 en masse-- P4 will always be a niche market relative to the programmable datapath space and the skill sets required to support serious scalable deployment. That being said, there will be a nontrivial contingent of users who need to run the same programs in both SW and HW environments. Expecting them to maintain two very different code bases to support two disparate models is costly and prohibitive to them. So for their benefit, we need a solution to reconcile these two models. P4TC is one means to accomplish that.

We want to consider both the permutations: 1) compile C code to run in
P4 hardware 2) compile P4 to run in SW. If we establish a common IR, then we can generalize the problem: programmer writes their datapath in the language of their choosing (P4, C, Python, Rust, etc.), they compile the program to whatever backend they are using (HW, SW, XDP/eBPF, etc.). The P4TC CLI serves as one such IR as there's nothing that prevents someone from compiling a program from another language to the CLI (for instance, we've implemented the compiler to output the parser CLI from PANDA-C). The CLI natively runs in kernel SW, and with the right hooks could be offloaded to HW-- not just P4 hardware but potentially other hardware targets as well.

Tom

>  
> https://www.intel.com/content/www/us/en/products/details/network-io/ip
> u/e2000-asic.html
> Here are some other useful links
> https://ipdk.io/
>
> Anjali
>
> -----Original Message-----
> From: Jamal Hadi Salim <hadi@...atatu.com>
> Sent: Friday, January 27, 2023 11:43 AM
> To: Jakub Kicinski <kuba@...nel.org>
> Cc: Jamal Hadi Salim <jhs@...atatu.com>; netdev@...r.kernel.org; 
> kernel@...atatu.com; Chatterjee, Deb <deb.chatterjee@...el.com>; 
> Singhai, Anjali <anjali.singhai@...el.com>; Limaye, Namrata 
> <namrata.limaye@...el.com>; khalidm@...dia.com; tom@...anda.io; 
> pratyush@...anda.io; jiri@...nulli.us; xiyou.wangcong@...il.com; 
> davem@...emloft.net; edumazet@...gle.com; pabeni@...hat.com; 
> vladbu@...dia.com; simon.horman@...igine.com; stefanc@...vell.com; 
> seong.kim@....com; mattyk@...dia.com; Daly, Dan <dan.daly@...el.com>; 
> Fingerhut, John Andy <john.andy.fingerhut@...el.com>
> Subject: Re: [PATCH net-next RFC 00/20] Introducing P4TC
>
> On Fri, Jan 27, 2023 at 12:18 PM Jakub Kicinski <kuba@...nel.org> wrote:
> >
> > On Fri, 27 Jan 2023 08:33:39 -0500 Jamal Hadi Salim wrote:
> > > On Thu, Jan 26, 2023 at 6:30 PM Jakub Kicinski <kuba@...nel.org> wrote:
> > > > On Tue, 24 Jan 2023 12:03:46 -0500 Jamal Hadi Salim wrote:
>
> [..]
> > > Network programmability involving hardware  - where at minimal the 
> > > specification of the datapath is in P4 and often the 
> > > implementation is. For samples of specification using P4 (that are 
> > > public) see for example MS Azure:
> > > https://github.com/sonic-net/DASH/tree/main/dash-pipeline
> >
> > That's an IPU thing?
> >
>
> Yes, DASH is xPU. But the whole Sonic/SAI thing includes switches and P4 plays a role there.
>
> > > If you are a vendor and want to sell a NIC in that space, the spec 
> > > you get is in P4.
> >
> > s/NIC/IPU/ ?
>
> I do believe that one can write a P4 program to express things a regular NIC could express that may be harder to expose with current interfaces.
>
> > > Your underlying hardware
> > > doesnt have to be P4 native, but at minimal the abstraction (as we 
> > > are trying to provide with P4TC) has to be able to consume the P4 
> > > specification.
> >
> > P4 is certainly an option, especially for specs, but I haven't seen 
> > much adoption myself.
>
> The xPU market outside of hyper-scalers is emerging now. Hyperscalers looking at xPUs are looking at P4 as the datapath language - that sets the trend forward to large enterprises.
> That's my experience.
> Some of the vendors on the Cc should be able to point to adoption.
> Anjali? Matty?
>
> > What's the benefit / use case?
>
> Of P4 or xPUs?
> Unified approach to standardize how a datapath is defined is a value for P4.
> Providing a singular abstraction via the kernel (as opposed to every vendor pitching their API) is what the kernel brings.
>
> > > For implementations where P4 is in use, there are many - some 
> > > public others not, sample space:
> > > https://cloud.google.com/blog/products/gcp/google-cloud-using-p4ru
> > > nt
> > > ime-to-build-smart-networks
> >
> > Hyper-scaler proprietary.
>
> The control abstraction (P4 runtime) is certainly not proprietary.
> The datapath that is targetted by the runtime is.
> Hopefully we can fix that with P4TC.
> The majority of the discussions i have with some of the folks who do kernel bypass have one theme in common:
> The kernel process is just too long. Trying to add one feature to flower could take anywhere from 6 months to 3 years to finally show up in some supported distro. With P4TC we are taking the approach of scriptability to allow for speacilized datapaths (which P4 excels in). The google datapath maybe proprietary while their hardware may even(or not) be using native P4 - but the important detail is we have _a way_ to abstract those datapaths.
>
> > > There are NICs and switches which are P4 native in the market.
> >
> > Link to docs?
> >
>
> Off top of my head Intel Mount Evans, Pensando, Xilinx FPGAs, etc. The point is to bring them together under the linux umbrella.
>
> > > IOW, there is beacoup $ investment in this space that makes it worth pursuing.
> >
> > Pursuing $ is good! But the community IMO should maximize a 
> > different function.
>
> While I agree $ is not the primary motivator it is a factor, it is a good indicator. No different than the network stack being tweaked to do certain things that certain hyperscalers need because they invest $.
> I have no problems with a large harmonious tent.
>
> cheers,
> jamal
>
> > > TC is the kernel offload mechanism that has gathered deployment 
> > > experience over many years - hence P4TC.
> >
> > I don't wanna argue. I thought it'd be more fair towards you if I 
> > made my lack of conviction known, rather than sit quiet and ignore 
> > it since it's just an RFC.

Powered by blists - more mailing lists