Date:   Thu, 21 Jan 2021 12:59:55 -0800
From:   "Samudrala, Sridhar" <sridhar.samudrala@...el.com>
To:     Saeed Mahameed <saeed@...nel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Jason Gunthorpe <jgg@...dia.com>
Cc:     netdev@...r.kernel.org, linux-rdma@...r.kernel.org,
        alexander.duyck@...il.com, edwin.peer@...adcom.com,
        dsahern@...nel.org, kiran.patil@...el.com,
        jacob.e.keller@...el.com, david.m.ertman@...el.com,
        dan.j.williams@...el.com, Parav Pandit <parav@...dia.com>,
        Saeed Mahameed <saeedm@...dia.com>
Subject: Re: [net-next V9 14/14] net/mlx5: Add devlink subfunction port
 documentation

On 1/21/2021 12:52 AM, Saeed Mahameed wrote:
> From: Parav Pandit <parav@...dia.com>
>
> Add documentation for subfunction management using devlink
> port.
>
> Signed-off-by: Parav Pandit <parav@...dia.com>
> Signed-off-by: Saeed Mahameed <saeedm@...dia.com>
> ---
>   .../device_drivers/ethernet/mellanox/mlx5.rst | 210 ++++++++++++++++++
>   1 file changed, 210 insertions(+)
>
> diff --git a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> index a5eb22793bb9..a1b32fcd0d76 100644
> --- a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> +++ b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> @@ -12,6 +12,8 @@ Contents
>   - `Enabling the driver and kconfig options`_
>   - `Devlink info`_
>   - `Devlink parameters`_
> +- `mlx5 subfunction`_
> +- `mlx5 port function`_
>   - `Devlink health reporters`_
>   - `mlx5 tracepoints`_
>   
> @@ -181,6 +183,214 @@ User command examples:
>         values:
>            cmode driverinit value true
>   
> +mlx5 subfunction
> +================
> +mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
> +
> +A Subfunction has its own function capabilities and its own resources. This
> +means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
> +queues are neither shared nor stolen from the parent PCI function.
> +
> +When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
> +resources neither shared nor stolen from the parent PCI function.
> +
> +A subfunction has a dedicated window in PCI BAR space that is not shared
> +with ther other subfunctions or the parent PCI function. This ensures that all
> +devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
> +PCI BAR space.
> +
> +A Subfunction supports eswitch representation through which it supports tc
> +offloads. The user configures eswitch to send/receive packets from/to
> +the subfunction port.
> +
> +Subfunctions share PCI level resources such as PCI MSI-X IRQs with
> +other subfunctions and/or with its parent PCI function.
> +
> +Example mlx5 software, system and device view::
> +
> +       _______
> +      | admin |
> +      | user  |----------
> +      |_______|         |
> +          |             |
> +      ____|____       __|______            _________________
> +     |         |     |         |          |                 |
> +     | devlink |     | tc tool |          |    user         |
> +     | tool    |     |_________|          | applications    |
> +     |_________|         |                |_________________|
> +           |             |                   |          |
> +           |             |                   |          |         Userspace
> + +---------|-------------|-------------------|----------|--------------------+
> +           |             |           +----------+   +----------+   Kernel
> +           |             |           |  netdev  |   | rdma dev |
> +           |             |           +----------+   +----------+
> +   (devlink port add/del |              ^               ^
> +    port function set)   |              |               |
> +           |             |              +---------------|
> +      _____|___          |              |        _______|_______
> +     |         |         |              |       | mlx5 class    |
> +     | devlink |   +------------+       |       |   drivers     |
> +     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
> +     |_________|   +------------+       |       |_______________|
> +           |             |              |               ^
> +   (devlink ops)         |              |          (probe/remove)
> +  _________|________     |              |           ____|________
> + | subfunction      |    |     +---------------+   | subfunction |
> + | management driver|-----     | subfunction   |---|  driver     |
> + | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
> + |__________________|          +---------------+   |_____________|
> +           |                                            ^
> +  (sf add/del, vhca events)                             |
> +           |                                      (device add/del)
> +      _____|____                                    ____|________
> +     |          |                                  | subfunction |
> +     |  PCI NIC |---- activate/deactive events---->| host driver |
> +     |__________|                                  | (mlx5_core) |
> +                                                   |_____________|
> +
> +Subfunction is created using devlink port interface.
> +
> +- Change device to switchdev mode::
> +
> +    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
> +
> +- Add a devlink port of subfunction flaovur::
> +
> +    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
> +    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +      function:
> +        hw_addr 00:00:00:00:00:00 state inactive opstate detached
> +
> +- Show a devlink port of the subfunction::
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:00:00 state inactive opstate detached
> +
> +- Delete a devlink port of subfunction after use::
> +
> +    $ devlink port del pci/0000:06:00.0/32768
> +
> +mlx5 function attributes
> +========================
> +The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
> +a unified way for SmartNIC and non-SmartNIC.
> +
> +This is supported only when the eswitch mode is set to switchdev. Port function
> +configuration of the PCI VF/SF is supported through devlink eswitch port.
> +
> +Port function attributes should be set before PCI VF/SF is enumerated by the
> +driver.
> +
> +MAC address setup
> +-----------------
> +mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.
> +
> +The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
> +device created for the PCI VF/SF.
> +
> +- Get the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port show pci/0000:06:00.0/2
> +    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
> +      function:
> +        hw_addr 00:00:00:00:00:00
> +
> +- Set the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55
> +
> +    $ devlink port show pci/0000:06:00.0/2
> +    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
> +      function:
> +        hw_addr 00:11:22:33:44:55
> +
> +- Get the MAC address of the SF identified by its unique devlink port index::
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:00:00
> +
> +- Set the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:88:88
> +
> +SF state setup
> +--------------
> +To use the SF, the user must active the SF using the SF function state
> +attribute.
> +
> +- Get the state of the SF identified by its unique devlink port index::
> +
> +   $ devlink port show ens2f0npf0sf88
> +   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +     function:
> +       hw_addr 00:00:00:00:88:88 state inactive opstate detached
> +
> +- Activate the function and verify its state is active::
> +
> +   $ devlink port function set ens2f0npf0sf88 state active
> +
> +   $ devlink port show ens2f0npf0sf88
> +   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +     function:
> +       hw_addr 00:00:00:00:88:88 state active opstate detached
> +
> +Upon function activation, the PF driver instance gets the event from the device
> +that a particular SF was activated. It's the cue to put the device on bus, probe
> +it and instantiate the devlink instance and class specific auxiliary devices
> +for it.
> +
> +- Show the auxiliary device and port of the subfunction::
> +
> +    $ devlink dev show
> +    devlink dev show auxiliary/mlx5_core.sf.4
> +
> +    $ devlink port show auxiliary/mlx5_core.sf.4/1
> +    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
> +
> +    $ rdma link show mlx5_0/1
> +    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
> +
> +    $ rdma dev show
> +    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
> +    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
> +
> +- Subfunction auxiliary device and class device hierarchy::
> +
> +                 mlx5_core.sf.4
> +          (subfunction auxiliary device)
> +                       /\
> +                      /  \
> +                     /    \
> +                    /      \
> +                   /        \
> +      mlx5_core.eth.4     mlx5_core.rdma.4
> +     (sf eth aux dev)     (sf rdma aux dev)
> +         |                      |
> +         |                      |
> +      p0sf88                  mlx5_0
> +     (sf netdev)          (sf rdma device)

This picture seems to indicate that when SF is activated, a sub function 
auxiliary device is created
and when a driver is bound to that sub function aux device and probed, 2 
additional auxiliary devices are created.
Is this correct? Are all these auxiliary devices seen on the same aux bus?
Why do we need another sf eth aux device?


> +
> +Additionally, the SF port also gets the event when the driver attaches to the
> +auxiliary device of the subfunction. This results in changing the operational
> +state of the function. This provides visiblity to the user to decide when is it
> +safe to delete the SF port for graceful termination of the subfunction.
> +
> +- Show the SF port operational state::
> +
> +    $ devlink port show ens2f0npf0sf88
> +    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +      function:
> +        hw_addr 00:00:00:00:88:88 state active opstate attached
> +
>   Devlink health reporters
>   ========================
>   

Powered by blists - more mailing lists