Message-ID: <CAJ+HfNjkv-tQhk8XLEK6pa-YgGKsN7xc326s=N=xcVsK_ri2Lw@mail.gmail.com>
Date:   Tue, 16 Apr 2019 20:23:34 +0200
From:   Björn Töpel <bjorn.topel@...il.com>
To:     Jonathan Lemon <jonathan.lemon@...il.com>
Cc:     Jesper Dangaard Brouer <brouer@...hat.com>,
        Jakub Kicinski <jakub.kicinski@...ronome.com>,
        Björn Töpel <bjorn.topel@...el.com>,
        Ilias Apalodimas <ilias.apalodimas@...aro.org>,
        Toke Høiland-Jørgensen <toke@...hat.com>,
        "Karlsson, Magnus" <magnus.karlsson@...el.com>,
        maciej.fijalkowski@...el.com, Jason Wang <jasowang@...hat.com>,
        Alexei Starovoitov <ast@...com>,
        Daniel Borkmann <borkmann@...earbox.net>,
        John Fastabend <john.fastabend@...il.com>,
        David Miller <davem@...emloft.net>,
        Andy Gospodarek <andy@...yhouse.net>,
        Netdev <netdev@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        Thomas Graf <tgraf@...g.ch>,
        Thomas Monjalon <thomas@...jalon.net>
Subject: Re: Per-queue XDP programs, thoughts

On Tue, 16 Apr 2019 at 18:53, Jonathan Lemon <jonathan.lemon@...il.com> wrote:
>
> On 16 Apr 2019, at 6:55, Jesper Dangaard Brouer wrote:
>
> > On Mon, 15 Apr 2019 15:49:32 -0700
> > Jakub Kicinski <jakub.kicinski@...ronome.com> wrote:
> >
> >> On Mon, 15 Apr 2019 18:32:58 +0200, Jesper Dangaard Brouer wrote:
> >>> On Mon, 15 Apr 2019 13:59:03 +0200 Björn Töpel
> >>> <bjorn.topel@...el.com> wrote:
> >>>> Hi,
> >>>>
> >>>> As you probably can derive from the amount of time this is taking,
> >>>> I'm
> >>>> not really satisfied with the design of per-queue XDP program.
> >>>> (That,
> >>>> plus I'm a terribly slow hacker... ;-)) I'll try to expand my
> >>>> thinking
> >>>> in this mail!
> >>
> >> Jesper was advocating per-queue progs since very early days of XDP.
> >> If it was easy to implement cleanly we would've already gotten it ;)
> >
> > (I cannot help to feel offended here...  IMHO that statement is BS,
> > that is not how upstream development work, and sure, I am to blame as
> > I've simply been to lazy or busy with other stuff to implement it.  It
> > is not that hard to send down a queue# together with the XDP attach
> > command.)
> >
> > I've been advocating for per-queue progs from day-1, since this is an
> > obvious performance advantage, given the programmer can specialize the
> > BPF/XDP-prog to the filtered traffic.  I hope/assume we are on the
> > same
> > pages here, that per-queue progs is a performance optimization.
> >
> > I guess the rest of the discussion in this thread is (1) if we can
> > convince each-other that someone will actually use this optimization,
> > and (2) if we can abstract this away from the user.
> >
> >
> >>>> Beware, it's kind of a long post, and it's all over the place.
> >>>
> >>> Cc'ing all the XDP-maintainers (and netdev).
> >>>
> >>>> There are a number of ways of setting up flows in the kernel, e.g.
> >>>>
> >>>> * Connecting/accepting a TCP socket (in-band)
> >>>> * Using tc-flower (out-of-band)
> >>>> * ethtool (out-of-band)
> >>>> * ...
> >>>>
> >>>> The first acts on sockets, the second on netdevs. Then there's
> >>>> ethtool
> >>>> to configure RSS, and the RSS-on-steriods rxhash/ntuple that can
> >>>> steer
> >>>> to queues. Most users care about sockets and netdevices. Queues is
> >>>> more of an implementation detail of Rx or for QoS on the Tx side.
> >>>
> >>> Let me first acknowledge that the current Linux tools to
> >>> administrator
> >>> HW filters is lacking (well sucks).  We know the hardware is
> >>> capable,
> >>> as DPDK have an full API for this called rte_flow[1]. If nothing
> >>> else
> >>> you/we can use the DPDK API to create a program to configure the
> >>> hardware, examples here[2]
> >>>
> >>>  [1] https://doc.dpdk.org/guides/prog_guide/rte_flow.html
> >>>  [2] https://doc.dpdk.org/guides/howto/rte_flow.html
> >>>
> >>>> XDP is something that we can attach to a netdevice. Again, very
> >>>> natural from a user perspective. As for XDP sockets, the current
> >>>> mechanism is that we attach to an existing netdevice queue. Ideally
> >>>> what we'd like is to *remove* the queue concept. A better approach
> >>>> would be creating the socket and set it up -- but not binding it to
> >>>> a
> >>>> queue. Instead just binding it to a netdevice (or crazier just
> >>>> creating a socket without a netdevice).
> >>
> >> You can remove the concept of a queue from the AF_XDP ABI (well,
> >> extend
> >> it to not require the queue being explicitly specified..), but you
> >> can't
> >> avoid the user space knowing there is a queue.  Because if you do you
> >> can no longer track and configure that queue (things like IRQ
> >> moderation, descriptor count etc.)
> >
> > Yes exactly.  Bjørn you mentioned leaky abstractions, and by removing
> > the concept of a queue# from the AF_XDP ABI, then you have basically
> > created a leaky abstraction, as the sysadm would need to
> > tune/configure
> > the "hidden" abstracted queue# (IRQ moderation, desc count etc.).
> >
> >> Currently the term "queue" refers mostly to the queues that stack
> >> uses.
> >> Which leaves e.g. the XDP TX queues in a strange grey zone (from
> >> ethtool channel ABI perspective, RPS, XPS etc.)  So it would be nice
> >> to
> >> have the HW queue ID somewhat detached from stack queue ID.  Or at
> >> least
> >> it'd be nice to introduce queue types?  I've been pondering this for
> >> a
> >> while, I don't see any silver bullet here..
> >
> > Yes! - I also worry about the term "queue".  This is very interesting
> > to discuss.
> >
> > I do find it very natural that your HW (e.g. Netronome) have several
> > HW
> > RX-queues that feed/send to a single software NAPI RX-queue.  (I
> > assume
> > this is how you HW already works, but software/Linux cannot know this
> > internal HW queue id).  How we expose and use this is interesting.
> >
> > I do want to be-able to create new RX-queues, semi-dynamically
> > "setup"/load time.  But still a limited number of RX-queues, for
> > performance and memory reasons (TX-queue are cheaper).  Memory as we
> > prealloc memory RX-queue (and give it to HW).  Performance as with too
> > many queues, there is less chance to have a (RX) bulk of packets in
> > queue.
>
> How would these be identified?  Suppose there's a set of existing RX
> queues for the device which handle the normal system traffic - then I
> add an AF_XDP socket which gets its own dedicated RX queue.  Does this
> create a new queue id for the device?  Create a new namespace with its
> own queue id?
>
> The entire reason the user even cares about the queue id at all is
> because it needs to use ethtool/netlink/tc for configuration, or the
> net device's XDP program needs to differentiate between the queues
> for specific treatment.
>

Exactly!

I've been thinking along these lines as well -- I'd like to go the
torwards a "AF_XDP with dedicated queues" model (in addition to the
attach one). Then again, as Jesper and Jakub reminded me, XDP Tx is
yet another inaccessible (from a configuration standpoint) set of
queues. Maybe there is a need for proper "queues". Some are attached
to the kernel stack, some to XDP Tx and some to AF_XDP sockets.

That said, I like the good old netdevice and socket model for user
applications. "There's a bunch of pipes. Some have HW backing, but I
don't care much." :-P


Björn

> >
> > For example I would not create an RX-queue per TCP-flow.  But why do I
> > still want per-queue XDP-progs and HW-filters for this TCP-flow
> > use-case... let me explain:
> >
> >   E.g. I want to implement an XDP TCP socket load-balancer (same host
> > delivery, between XDP and network stack).  And my goal is to avoid
> > touching packet payload on XDP RX-CPU.  First I configure ethtool
> > filter to redirect all TCP port 80 to a specific RX-queue (could also
> > be N-queues), then I don't need to parse TCP-port-80 in my per-queue
> > BPF-prog, and I have higher chance of bulk-RX.  Next I need HW to
> > provide some flow-identifier, e.g. RSS-hash, flow-id or internal
> > HW-queue-id, which I can use to redirect on (e.g. via CPUMAP to
> > N-CPUs).  This way I don't touch packet payload on RX-CPU (my bench
> > shows one RX-CPU can handle between 14-20Mpps).
> >
> >
> >>> Let me just remind everybody that the AF_XDP performance gains comes
> >>> from binding the resource, which allow for lock-free semantics, as
> >>> explained here[3].
> >>>
> >>> [3]
> >>> https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP#where-does-af_xdp-performance-come-from
> >>>
> >>>
> >>>> The socket is an endpoint, where I'd like data to end up (or get
> >>>> sent
> >>>> from). If the kernel can attach the socket to a hardware queue,
> >>>> there's zerocopy if not, copy-mode. Dito for Tx.
> >>>
> >>> Well XDP programs per RXQ is just a building block to achieve this.
> >>>
> >>> As Van Jacobson explain[4], sockets or applications "register" a
> >>> "transport signature", and gets back a "channel".   In our case, the
> >>> netdev-global XDP program is our way to register/program these
> >>> transport
> >>> signatures and redirect (e.g. into the AF_XDP socket).
> >>> This requires some work in software to parse and match transport
> >>> signatures to sockets.  The XDP programs per RXQ is a way to get
> >>> hardware to perform this filtering for us.
> >>>
> >>>  [4] http://www.lemis.com/grog/Documentation/vj/lca06vj.pdf
> >>>
> >>>
> >>>> Does a user (control plane) want/need to care about queues? Just
> >>>> create a flow to a socket (out-of-band or inband) or to a netdevice
> >>>> (out-of-band).
> >>>
> >>> A userspace "control-plane" program, could hide the setup and use
> >>> what
> >>> the system/hardware can provide of optimizations.  VJ[4] e.g.
> >>> suggest
> >>> that the "listen" socket first register the transport signature
> >>> (with
> >>> the driver) on "accept()".   If the HW supports DPDK-rte_flow API we
> >>> can register a 5-tuple (or create TC-HW rules) and load our
> >>> "transport-signature" XDP prog on the queue number we choose.  If
> >>> not,
> >>> when our netdev-global XDP prog need a hash-table with 5-tuple and
> >>> do
> >>> 5-tuple parsing.
> >>
> >> But we do want the ability to configure the queue, and get stats for
> >> that queue.. so we can't hide the queue completely, right?
> >
> > Yes, that is yet another example that the queue id "leak".
> >
> >
> >>> Creating netdevices via HW filter into queues is an interesting
> >>> idea.
> >>> DPDK have an example here[5], on how to per flow (via ethtool filter
> >>> setup even!) send packets to queues, that endup in SRIOV devices.
> >>>
> >>>  [5] https://doc.dpdk.org/guides/howto/flow_bifurcation.html
> >>
> >> I wish I had the courage to nack the ethtool redirect to VF Intel
> >> added :)
> >>
> >>>> Do we envison any other uses for per-queue XDP other than AF_XDP?
> >>>> If
> >>>> not, it would make *more* sense to attach the XDP program to the
> >>>> socket (e.g. if the endpoint would like to use kernel data
> >>>> structures
> >>>> via XDP).
> >>>
> >>> As demonstrated in [5] you can use (ethtool) hardware filters to
> >>> redirect packets into VFs (Virtual Functions).
> >>>
> >>> I also want us to extend XDP to allow for redirect from a PF
> >>> (Physical
> >>> Function) into a VF (Virtual Function).  First the netdev-global
> >>> XDP-prog need to support this (maybe extend xdp_rxq_info with PF +
> >>> VF
> >>> info).  Next configure HW filter to queue# and load XDP prog on that
> >>> queue# that only "redirect" to a single VF.  Now if driver+HW
> >>> supports
> >>> it, it can "eliminate" the per-queue XDP-prog and do everything in
> >>> HW.
> >>
> >> That sounds slightly contrived.  If the program is not doing
> >> anything,
> >> why involve XDP at all?
> >
> > If the HW doesn't support this then the XDP software will do the work.
> > If the HW supports this, then you can still list the XDP-prog via
> > bpftool, and see that you have a XDP prog that does this action (and
> > maybe expose a offloaded-to-HW bit if you like to expose this info).
> >
> >
> >>  As stated above we already have too many ways
> >> to do flow config and/or VF redirect.
> >>
> >>>> If we'd like to slice a netdevice into multiple queues. Isn't
> >>>> macvlan
> >>>> or similar *virtual* netdevices a better path, instead of
> >>>> introducing
> >>>> yet another abstraction?
> >>
> >> Yes, the question of use cases is extremely important.  It seems
> >> Mellanox is working on "spawning devlink ports" IOW slicing a device
> >> into subdevices.  Which is a great way to run bifurcated DPDK/netdev
> >> applications :/  If that gets merged I think we have to recalculate
> >> what purpose AF_XDP is going to serve, if any.
> >>
> >> In my view we have different "levels" of slicing:
> >
> > I do appreciate this overview of NIC slicing, as HW-filters +
> > per-queue-XDP can be seen as a way to slice up the NIC.
> >
> >>  (1) full HW device;
> >>  (2) software device (mdev?);
> >>  (3) separate netdev;
> >>  (4) separate "RSS instance";
> >>  (5) dedicated application queues.
> >>
> >> 1 - is SR-IOV VFs
> >> 2 - is software device slicing with mdev (Mellanox)
> >> 3 - is (I think) Intel's VSI debugfs... "thing"..
> >> 4 - is just ethtool RSS contexts (Solarflare)
> >> 5 - is currently AF-XDP (Intel)
> >>
> >> (2) or lower is required to have raw register access allowing
> >> vfio/DPDK
> >> to run "natively".
> >>
> >> (3) or lower allows for full reuse of all networking APIs, with very
> >> natural RSS configuration, TC/QoS configuration on TX etc.
> >>
> >> (5) is sufficient for zero copy.
> >>
> >> So back to the use case.. seems like AF_XDP is evolving into allowing
> >> "level 3" to pass all frames directly to the application?  With
> >> optional XDP filtering?  It's not a trick question - I'm just trying
> >> to
> >> place it somewhere on my mental map :)
> >
> >
> >>> XDP redirect a more generic abstraction that allow us to implement
> >>> macvlan.  Except macvlan driver is missing ndo_xdp_xmit. Again first
> >>> I
> >>> write this as global-netdev XDP-prog, that does a lookup in a
> >>> BPF-map.
> >>> Next I configure HW filters that match the MAC-addr into a queue#
> >>> and
> >>> attach simpler XDP-prog to queue#, that redirect into macvlan
> >>> device.
> >>>
> >>>> Further, is queue/socket a good abstraction for all devices? Wifi?
> >>
> >> Right, queue is no abstraction whatsoever.  Queue is a low level
> >> primitive.
> >
> > I agree, queue is a low level primitive.
> >
> > This the basically interface that the NIC hardware gave us... it is
> > fairly limited as it can only express a queue id and a IRQ line that
> > we
> > can try to utilize to scale the system.   Today, we have not really
> > tapped into the potential of using this... instead we simply RSS-hash
> > balance across all RX-queues and hope this makes the system scale...
> >
> >
> >>>> By just viewing sockets as an endpoint, we leave it up to the
> >>>> kernel to
> >>>> figure out the best way. "Here's an endpoint. Give me data
> >>>> **here**."
> >>>>
> >>>> The OpenFlow protocol does however support the concept of queues
> >>>> per
> >>>> port, but do we want to introduce that into the kernel?
> >>
> >> Switch queues != host queues.  Switch/HW queues are for QoS, host
> >> queues
> >> are for RSS.  Those two concepts are similar yet different.  In Linux
> >> if you offload basic TX TC (mq)prio (the old work John has done for
> >> Intel) the actual number of HW queues becomes "channel count" x "num
> >> TC
> >> prios".  What would queue ID mean for AF_XDP in that setup, I wonder.
> >
> > Thanks for explaining that. I must admit I never really understood the
> > mqprio concept and these "prios" (when reading the code and playing
> > with it).
> >
> >
> >>>> So, if per-queue XDP programs is only for AF_XDP, I think it's
> >>>> better
> >>>> to stick the program to the socket. For me per-queue is sort of a
> >>>> leaky abstraction...
> >>>>
> >>>> More thoughts. If we go the route of per-queue XDP programs. Would
> >>>> it
> >>>> be better to leave the setup to XDP -- i.e. the XDP program is
> >>>> controlling the per-queue programs (think tail-calls, but a map
> >>>> with
> >>>> per-q programs). Instead of the netlink layer. This is part of a
> >>>> bigger discussion, namely should XDP really implement the control
> >>>> plane?
> >>>>
> >>>> I really like that a software switch/router can be implemented
> >>>> effectively with XDP, but ideally I'd like it to be offloaded by
> >>>> hardware -- using the same control/configuration plane. Can we do
> >>>> it
> >>>> in hardware, do that. If not, emulate via XDP.
> >>
> >> There is already a number of proposals in the "device application
> >> slicing", it would be really great if we could make sure we don't
> >> repeat the mistakes of flow configuration APIs, and try to prevent
> >> having too many of them..
> >>
> >> Which is very challenging unless we have strong use cases..
> >>
> >>> That is actually the reason I want XDP per-queue, as it is a way to
> >>> offload the filtering to the hardware.  And if the per-queue
> >>> XDP-prog
> >>> becomes simple enough, the hardware can eliminate and do everything
> >>> in
> >>> hardware (hopefully).
> >>>
> >>>> The control plane should IMO be outside of the XDP program.
> >>
> >> ENOCOMPUTE :)  XDP program is the BPF byte code, it's never control
> >> plane.  Do you mean application should not control the "context/
> >> channel/subdev" creation?  You're not saying "it's not the XDP
> >> program
> >> which should be making the classification", no?  XDP program
> >> controlling the classification was _the_ reason why we liked AF_XDP
> >> :)
> >
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   LinkedIn: http://www.linkedin.com/in/brouer

Powered by blists - more mailing lists