Message-ID: <CAM0EoMkV6STz+aGuZ8vtHHXSkS0yQJpcre32B5erLbqRiY6Azg@mail.gmail.com>
Date:   Thu, 16 Feb 2023 15:24:01 -0500
From:   Jamal Hadi Salim <jhs@...atatu.com>
To:     Jamal Hadi Salim <hadi@...atatu.com>
Cc:     Edward Cree <ecree.xilinx@...il.com>,
        Toke Høiland-Jørgensen <toke@...hat.com>,
        Jiri Pirko <jiri@...nulli.us>,
        John Fastabend <john.fastabend@...il.com>,
        Willem de Bruijn <willemb@...gle.com>,
        Stanislav Fomichev <sdf@...gle.com>,
        Jakub Kicinski <kuba@...nel.org>, netdev@...r.kernel.org,
        kernel@...atatu.com, deb.chatterjee@...el.com,
        anjali.singhai@...el.com, namrata.limaye@...el.com,
        khalidm@...dia.com, tom@...anda.io, pratyush@...anda.io,
        xiyou.wangcong@...il.com, davem@...emloft.net, edumazet@...gle.com,
        pabeni@...hat.com, vladbu@...dia.com, simon.horman@...igine.com,
        stefanc@...vell.com, seong.kim@....com, mattyk@...dia.com,
        dan.daly@...el.com, john.andy.fingerhut@...el.com,
        "Jain, Vipin" <Vipin.Jain@....com>
Subject: Re: [PATCH net-next RFC 00/20] Introducing P4TC

Hi,

Want to provide an update to this thread and a summary of where we are
(typing this on web browser client so i hope it doesnt come all
mangled up):

I have had high bandwidth discussions with several people offlist
(thanks to everyone who invested their time in trying to smoothen
this); sometimes cooler   headers prevail this way. We are willing
(and are starting) to invest time to  see how we can fit ebpf for the
software datapath. Should be noted that we did  look at ebpf when this
project started and we ended up not going that path. I think what is
new in this equation is the concept of kfuncs - which we didnt  have
back then. Perhaps with kfuncs we can make both worlds work together.
XDP as well is appealing.

As i have stated earlier:
The starting premise is that the posted code meets our requirements so
 whatever we do using ebpf has to meet our requirements. I am ok with
some limited degree of square hole round peg situation but it cant  be
interfering in meeting our goals.

So let me restate those goals so we dont go into some rabit hole in
the discussion:
1) Supporting P4 in the kernel both for the sw and hw datapath
utilizing the well established tc infra which allows both sw
equivalence and hw offload.  We are _not_ going to reinvent this.
Essentially we get the whole package: from the control plane to the
tooling infra, netlink messaging to s/w and h/w symbiosis, the
autonomous kernel control, etc. The advantage is that we have a
singular vendor-neutral interface via the kernel using well understood
mechanisms.
Behavioral equivalence between hw and sw is a given.

2) Operational usability - this is encoded currently in the
scriptability approach. Ex, I can just ship someone a shell script in
an email but more important if they have deployed tc offloads the
runtime semantics are unchanged. "write once, run anywhere" paradigm
is easier to state in ascii;-> The interface is designed to be
scriptable to remove the burden of  making kernel and user space code
changes for any new processing functions  (whether in s/w or
hardware).
3) Debuggability - developers and ops people who are familiar with tc
offloads  can continue using the _same existing techniques and tools_.
This also eases support.

4) Performance - note our angle on this, based on the niche we are
looking at  is "if you want performance then offload". However, one
discussion point that has been raised multiple times in the thread and
in private is that  there are performance gains when using ebpf. This
arguement is reasonable and a motivator for us to invest our time in
evaluating.

 We have started doing off the cuff measurements. With very simple P4
program which receives a packet, looks up a table, and on a hit
changes src/mac address then forwards. We have: A) implemented a
handcoded ebpf program, B) generated  P4TC sw only C) flower s/w  only
(skip_hw) rules and D) hardware offload (skip_sw) all on tc (so we can
do orange-orange comparison). The SUT has a dual port CX6 NIC capable
of offloading pedit and mirred. Trex is connected to one port and
sending http gets which goes via the box and a response comes back on
the other port which we send back to trex. The traffic is very
asymettric; data coming  back to the client fills up the 25G pipe but
ACKs going back consume a lot less. Unfortunately all 4 scenarios were
able to handle the wire rate - we are going to set up more nasty
traffic generation later; for now we opted to look at cpu utilization
for the 4 scenarios. We have the following results:

 A) 35% CPU B) 39% C) 36% D) 0%

This is by no means a good test but i wanted to illustrate the
relevance of  #D (0%) - which is a main itch for us.

We need to test more complex programs which is where probably the
performance of ebpf will shine. XDP for sure will beat all the others
- but i would rather get the facts in place first. So we are investing
effort in this direction
and will share results at some point.

There may be other low hanging fruits that have been brought up in the
discussion for ebpf (the parser being one); we will be looking at all
those as well.

Note:
 The goal of this exercise for us is to evaluate not just performance
but   also consider how it affects the other P4TC goals. There may be
a sweet happy point somewhere in there but we need to collect the data
instead of hypothesizing.

cheers,
jamal

Powered by blists - more mailing lists