Message-ID: <5edbd360-7afb-2605-21ba-7337be15e235@nvidia.com>
Date:   Thu, 18 Aug 2022 18:07:34 +0300
From:   Roi Dayan <roid@...dia.com>
To:     ecree@...inx.com, netdev@...r.kernel.org, linux-net-drivers@....com
Cc:     davem@...emloft.net, kuba@...nel.org, pabeni@...hat.com,
        edumazet@...gle.com, corbet@....net, linux-doc@...r.kernel.org,
        jacob.e.keller@...el.com, jesse.brandeburg@...el.com,
        michael.chan@...adcom.com, andy@...yhouse.net, saeed@...nel.org,
        jiri@...nulli.us, snelson@...sando.io, simon.horman@...igine.com,
        alexander.duyck@...il.com, rdunlap@...radead.org,
        Edward Cree <ecree.xilinx@...il.com>
Subject: Re: [RFC PATCH v2 net-next] docs: net: add an explanation of VF (and
 other) Representors



On 2022-08-15 5:22 PM, ecree@...inx.com wrote:
> From: Edward Cree <ecree.xilinx@...il.com>
> 
> There's no clear explanation of what VF Representors are for, their
>   semantics, etc., outside of vendor docs and random conference slides.
> Add a document explaining Representors and defining what drivers that
>   implement them are expected to do.
> 
> Signed-off-by: Edward Cree <ecree.xilinx@...il.com>
> ---
> Changed in v2:
>   - incorporated feedback from Jakub including rewrite of the Motivation section,
>     representors for uplink/phys port, replace phys_port_name conventions with
>     devlink port.
>   - fixed archaic spelling (Randy)
>   - painted the bike shed blue ("master PF") for now, we can always change it
>     again later
>   - added Definitions section
> 
>   Documentation/networking/index.rst        |   1 +
>   Documentation/networking/representors.rst | 228 ++++++++++++++++++++++
>   Documentation/networking/switchdev.rst    |   1 +
>   3 files changed, 230 insertions(+)
>   create mode 100644 Documentation/networking/representors.rst
> 
> diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
> index 03b215bddde8..c37ea2b54c29 100644
> --- a/Documentation/networking/index.rst
> +++ b/Documentation/networking/index.rst
> @@ -93,6 +93,7 @@ Contents:
>      radiotap-headers
>      rds
>      regulatory
> +   representors
>      rxrpc
>      sctp
>      secid
> diff --git a/Documentation/networking/representors.rst b/Documentation/networking/representors.rst
> new file mode 100644
> index 000000000000..be7cc4752d11
> --- /dev/null
> +++ b/Documentation/networking/representors.rst
> @@ -0,0 +1,228 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +=============================
> +Network Function Representors
> +=============================
> +
> +This document describes the semantics and usage of representor netdevices, as
> +used to control internal switching on SmartNICs.  For the closely-related port
> +representors on physical (multi-port) switches, see
> +:ref:`Documentation/networking/switchdev.rst <switchdev>`.
> +
> +Motivation
> +----------
> +
> +Since the mid-2010s, network cards have started offering more complex
> +virtualisation capabilities than the legacy SR-IOV approach (with its simple
> +MAC/VLAN-based switching model) can support.  This led to a desire to offload
> +software-defined networks (such as OpenVSwitch) to these NICs to specify the
> +network connectivity of each function.  The resulting designs are variously
> +called SmartNICs or DPUs.
> +
> +Network function representors bring the standard Linux networking stack to
> +virtual switches and IOV devices.  Just as each port of a Linux-controlled
> +switch has a separate netdev, so each virtual function has one.  When the system
> +boots, and before any offload is configured, all packets from the virtual
> +functions appear in the networking stack of the PF via the representors.
> +The PF can thus always communicate freely with the virtual functions.
> +The PF can configure standard Linux forwarding between representors, the uplink
> +or any other netdev (routing, bridging, TC classifiers).
> +
> +Thus, a representor is both a control plane object (representing the function in
> +administrative commands) and a data plane object (one end of a virtual pipe).
> +As a virtual link endpoint, the representor can be configured like any other
> +netdevice; in some cases (e.g. link state) the representee will follow the
> +representor's configuration, while in others there are separate APIs to
> +configure the representee.
> +
> +Definitions
> +-----------
> +
> +This document uses the term "master PF" to refer to the PCIe function which has
> +administrative control over the virtual switch on the device.  Conceivably a NIC
> +could be configured to grant these administrative privileges instead to a VF or
> +SF (subfunction); the terminology is not meant to exclude this case.
> +Depending on NIC design, a multi-port NIC might have a single master PF for the
> +whole device or might have a separate virtual switch, and hence master PF, for
> +each physical network port.
> +If the NIC supports nested switching, there might be separate "master PFs" for
> +each nested switch, in which case each "master PF" should only create
> +representors for the ports on the (sub-)switch it directly administers.
> +
> +A "representee" is the object that a representor represents.  So for example in
> +the case of a VF representor, the representee is the corresponding VF.
> +
> +What does a representor do?
> +---------------------------
> +
> +A representor has three main roles.
> +
> +1. It is used to configure the network connection the representee sees, e.g.
> +   link up/down, MTU, etc.  For instance, bringing the representor
> +   administratively UP should cause the representee to see a link up / carrier
> +   on event.
> +2. It provides the slow path for traffic which does not hit any offloaded
> +   fast-path rules in the virtual switch.  Packets transmitted on the
> +   representor netdevice should be delivered to the representee; packets
> +   transmitted to the representee which fail to match any switching rule should
> +   be received on the representor netdevice.  (That is, there is a virtual pipe
> +   connecting the representor to the representee, similar in concept to a veth
> +   pair.)
> +   This allows software switch implementations (such as OpenVSwitch or a Linux
> +   bridge) to forward packets between representees and the rest of the network.
> +3. It acts as a handle by which switching rules (such as TC filters) can refer
> +   to the representee, allowing these rules to be offloaded.
> +
> +The combination of 2) and 3) means that the behaviour (apart from performance)
> +should be the same whether a TC filter is offloaded or not.  E.g. a TC rule
> +on a VF representor applies in software to packets received on that representor
> +netdevice, while in hardware offload it would apply to packets transmitted by
> +the representee VF.  Conversely, a mirred egress redirect to a VF representor
> +corresponds in hardware to delivery directly to the representee VF.
> +
> +What functions should have a representor?
> +-----------------------------------------
> +
> +Essentially, for each virtual port on the device's internal switch, there
> +should be a representor.
> +Some vendors have chosen to omit representors for the uplink and the physical
> +network port, which can simplify usage (the uplink netdev becomes in effect the
> +physical port's representor) but does not generalise to devices with multiple
> +ports or uplinks.
> +
> +Thus, the following should all have representors:
> +
> + - VFs belonging to the master PF.
> + - Other PFs on the local PCIe controller, and any VFs belonging to them.
> + - PFs and VFs on other PCIe controllers on the device (e.g. for any embedded
> +   System-on-Chip within the SmartNIC).
> + - PFs and VFs with other personalities, including network block devices (such
> +   as a vDPA virtio-blk PF backed by remote/distributed storage), if their
> +   network access is implemented through a virtual switch port.
> +   Note that such functions can require a representor despite the representee
> +   not having a netdev.
> + - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have
> +   their own port on the switch (as opposed to using their parent PF's port).
> + - Any accelerators or plugins on the device whose interface to the network is
> +   through a virtual switch port, even if they do not have a corresponding PCIe
> +   PF or VF.
> +
> +This allows the entire switching behaviour of the NIC to be controlled through
> +representor TC rules.
> +
> +A PCIe function which does not have network access through the internal switch
> +(not even indirectly through the hardware implementation of whatever services
> +the function provides) should *not* have a representor (even if it has a
> +netdev).
> +Such a function has no switch virtual port for the representor to configure or
> +to be the other end of the virtual pipe.
> +
> +How are representors created?
> +-----------------------------
> +
> +The driver instance attached to the master PF should enumerate the virtual ports
> +on the switch, and for each representee, create a pure-software netdevice which
> +has some form of in-kernel reference to the PF's own netdevice or driver private
> +data (``netdev_priv()``).
> +If switch ports can dynamically appear/disappear, the PF driver should create
> +and destroy representors appropriately.
> +The operations of the representor netdevice will generally involve acting
> +through the master PF.  For example, ``ndo_start_xmit()`` might send the packet
> +through a hardware TX queue attached to the master PF, with either packet
> +metadata or queue configuration marking it for delivery to the representee.
> +
> +How are representors identified?
> +--------------------------------
> +
> +The representor netdevice should *not* directly refer to a PCIe device (e.g.
> +through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the
> +representee or of the master PF.

Hi,
maybe I'm confused here, but why representor should not refer to pci
device ? it does exists today for systemd renaming.
and this is used beside of implementing the other ndos you mention
below.

     udevadm output after linking to PCI device:
     $ udevadm test-builtin net_id /sys/class/net/eth6
     Load module index
     Network interface NamePolicy= disabled on kernel command line, 
ignoring.
     Parsed configuration file /usr/lib/systemd/network/99-default.link
     Created link configuration context.
     Using default interface naming scheme 'v243'.
     ID_NET_NAMING_SCHEME=v243
     ID_NET_NAME_PATH=enp0s8f0npf0vf0
     Unload module index
     Unloaded link configuration context.

$  git grep SET_NETDEV_DEV|grep rep
drivers/net/ethernet/intel/ice/ice_repr.c: 
SET_NETDEV_DEV(repr->netdev, ice_pf_to_dev(vf->pf));
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c: 
SET_NETDEV_DEV(netdev, mdev->device);
drivers/net/ethernet/netronome/nfp/flower/main.c: 
SET_NETDEV_DEV(repr, &priv->nn->pdev->dev);


> +Instead, it should implement the ``ndo_get_devlink_port()`` netdevice op, which
> +the kernel uses to provide the ``phys_switch_id`` and ``phys_port_name`` sysfs
> +nodes.  (Some legacy drivers implement ``ndo_get_port_parent_id()`` and
> +``ndo_get_phys_port_name`` directly, but this is deprecated.)  See
> +:ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>` for the
> +details of this API.
> +
> +It is expected that userland will use this information (e.g. through udev rules)
> +to construct an appropriately informative name or alias for the netdevice.  For
> +instance if the master PF is ``eth4`` then a representor with a
> +``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``.
> +
> +There are as yet no established conventions for naming representors which do not
> +correspond to PCIe functions (e.g. accelerators and plugins).
> +
> +How do representors interact with TC rules?
> +-------------------------------------------
> +
> +Any TC rule on a representor applies (in software TC) to packets received by
> +that representor netdevice.  Thus, if the delivery part of the rule corresponds
> +to another port on the virtual switch, the driver may choose to offload it to
> +hardware, applying it to packets transmitted by the representee.
> +
> +Similarly, since a TC mirred egress action targeting the representor would (in
> +software) send the packet through the representor (and thus indirectly deliver
> +it to the representee), hardware offload should interpret this as delivery to
> +the representee.
> +
> +As a simple example, if ``eth0`` is the master PF's netdevice and ``eth1`` is a
> +VF representor, the following rules::
> +
> +    tc filter add dev eth1 parent ffff: protocol ipv4 flower \
> +        action mirred egress redirect dev eth0
> +    tc filter add dev eth0 parent ffff: protocol ipv4 flower \
> +        action mirred egress mirror dev eth1
> +
> +would mean that all IPv4 packets from the VF are sent out the physical port, and
> +all IPv4 packets received on the physical port are delivered to the VF in
> +addition to the master PF.
> +
> +Of course the rules can (if supported by the NIC) include packet-modifying
> +actions (e.g. VLAN push/pop), which should be performed by the virtual switch.
> +
> +Tunnel encapsulation and decapsulation are rather more complicated, as they
> +involve a third netdevice (a tunnel netdev operating in metadata mode, such as
> +a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and
> +require an IP address to be bound to the underlay device (e.g. master PF or port
> +representor).  TC rules such as::
> +
> +    tc filter add dev eth1 parent ffff: flower \
> +        action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \
> +                              dst_port 4789 \
> +        action mirred egress redirect dev vxlan0
> +    tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \
> +        enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \
> +        action tunnel_key unset action mirred egress redirect dev eth1
> +
> +where ``LOCAL_IP`` is an IP address bound to ``eth0``, and ``REMOTE_IP`` is
> +another IP address on the same subnet, mean that packets sent by the VF should
> +be VxLAN encapsulated and sent out the physical port (the driver has to deduce
> +this by a route lookup of ``LOCAL_IP`` leading to ``eth0``, and also perform an
> +ARP/neighbour table lookup to find the MAC addresses to use in the outer
> +Ethernet frame), while UDP packets received on the physical port with UDP port
> +4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``, decapsulated
> +and forwarded to the VF.
> +
> +If this all seems complicated, just remember the 'golden rule' of TC offload:
> +the hardware should ensure the same final results as if the packets were
> +processed through the slow path, traversed software TC and were transmitted or
> +received through the representor netdevices.
> +
> +Configuring the representee's MAC
> +---------------------------------
> +
> +The representee's link state is controlled through the representor.  Setting the
> +representor administratively UP or DOWN should cause carrier ON or OFF at the
> +representee.
> +
> +Setting an MTU on the representor should cause that same MTU to be reported to
> +the representee.
> +(On hardware that allows configuring separate and distinct MTU and MRU values,
> +the representor MTU should correspond to the representee's MRU and vice-versa.)
> +
> +Currently there is no way to use the representor to set the station permanent
> +MAC address of the representee; other methods available to do this include:
> +
> + - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``)
> + - devlink port function (see **devlink-port(8)** and
> +   :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`)
> diff --git a/Documentation/networking/switchdev.rst b/Documentation/networking/switchdev.rst
> index f1f4e6a85a29..21e80c8e661b 100644
> --- a/Documentation/networking/switchdev.rst
> +++ b/Documentation/networking/switchdev.rst
> @@ -1,5 +1,6 @@
>   .. SPDX-License-Identifier: GPL-2.0
>   .. include:: <isonum.txt>
> +.. _switchdev:
>   
>   ===============================================
>   Ethernet switch device driver model (switchdev)

Powered by blists - more mailing lists