Message-ID: <20190902072213.7683-1-saeedm@mellanox.com>
Date:   Mon, 2 Sep 2019 07:22:50 +0000
From:   Saeed Mahameed <saeedm@...lanox.com>
To:     "David S. Miller" <davem@...emloft.net>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Alex Vesker <valex@...lanox.com>,
        Erez Shitrit <erezsh@...lanox.com>,
        Saeed Mahameed <saeedm@...lanox.com>
Subject: [pull request][net-next 00/18] Mellanox, mlx5 software managed
 steering

Hi Dave,

This series adds the support for software (driver managed) flow steering.
For more information please see tag log below.

Please pull and let me know if there is any problem.

Please note that the series starts with a merge of mlx5-next branch,
to resolve and avoid dependency with rdma tree.

Thanks,
Saeed.

---
The following changes since commit a06ebb8d953b4100236f3057be51d67640e06323:

  Merge branch 'mlx5-next' of git://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux (2019-09-02 00:16:05 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2019-09-01

for you to fetch changes up to 6208aecc121dde491047008f8dbc1e734c8e634b:

  net/mlx5: Add devlink flow_steering_mode parameter (2019-09-02 00:16:14 -0700)

----------------------------------------------------------------
mlx5-updates-2019-09-01  (Software steering support)

Abstract:
--------
Mellanox ConnetX devices supports packet matching, packet modification and
redirection. These functionalities are also referred to as flow-steering.
To configure a steering rule, the rule is written to the device owned
memory, this memory is accessed and cached by the device when processing
a packet.
Steering rules are constructed from multiple steering entries (STE).

Rules are configured using the Firmware command interface. The Firmware
processes the given driver command and translates them to STEs, then
writes them to the device memory in the current steering tables.
This process is slow due to the architecture of the command interface and
the processing complexity of each rule.

The highlight of this patchset is to cut the middle man (The firmware) and
do steering rules programming into device directly from the driver, with
no firmware intervention whatsoever.

Motivation:
-----------
Software (driver managed) steering allows for high rule insertion rates
compared to the FW steering described above, this is achieved by using
internal RDMA writes to the device owned memory instead of the slow
command interface to program steering rules.

Software (driver managed) steering, doesn't depend on new FW
for new steering functionality, new implementations can be done in the
driver skipping the FW layer.

Performance:
------------
The insertion rate on a single core using the new approach allows
programming ~300K rules per sec.

Test: TC L2 rules
33K/s with Software steering (this patchset).
5K/s  with FW and current driver.
This will improve OVS based solution performance.

Architecture and implementation details:
----------------------------------------
Software steering will be dynamically selected via devlink device
parameter. Example:
$ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
          pci/0000:06:00.0:
          name flow_steering_mode type driver-specific
          values:
             cmode runtime value smfs

mlx5 software steering module a.k.a (DR - Direct Rule) is implemented
and contained in mlx5/core/steering directory and controlled by
MLX5_SW_STEERING kconfig flag.

mlx5 core steering layer (fs_core) already provides a shim layer for
implementing different steering mechanisms, software steering will
leverage that as seen at the end of this series.

When Software Steering for a specific steering domain
(NIC/RDMA/Vport/ESwitch, etc ..) is supported, it will cause rules
targeting this domain to be created using  SW steering instead of FW.

The implementation includes:
Domain - The steering domain is the object that all other object resides
    in. It holds the memory allocator, send engine, locks and other shared
    data needed by lower objects such as table, matcher, rule, action.
    Each domain can contain multiple tables. Domain is equivalent to
    namespaces e.g (NIC/RDMA/Vport/ESwitch, etc ..) as implemented
    currently in mlx5_core fs_core (flow steering core).

Table - Table objects are used for holding multiple matchers, each table
    has a level used to prevent processing loops. Packets are being
    directed to this table once it is set as the root table, this is done
    by fs_core using a FW command. A packet is being processed inside the
    table matcher by matcher until a successful hit, otherwise the packet
    will perform the default action.

Matcher - Matchers objects are used to specify the fields mask for
    matching when processing a packet. A matcher belongs to a table, each
    matcher can hold multiple rules, each rule with different matching
    values corresponding to the matcher mask. Each matcher has a priority
    used for rule processing order inside the table.

Action - Action objects are created to specify different steering actions
    such as count, reformat (encapsulate, decapsulate, ...), modify
    header, forward to table and many other actions. When creating a rule
    a sequence of actions can be provided to be executed on a successful
    match.

Rule - Rule objects are used to specify a specific match on packets as
    well as the actions that should be executed. A rule belongs to a
    matcher.

STE - This layer is used to hold the specific STE format for the device
    and to convert the requested rule to STEs. Each rule is constructed of
    an STE chain, Multiple rules construct a steering graph. Each node in
    the graph is a hash table containing multiple STEs. The index of each
    STE in the hash table is being calculated using a CRC32 hash function.

Memory pool - Used for managing and caching device owned memory for rule
    insertion. The memory is being allocated using DM (device memory) API.

Communication with device - layer for standard RDMA operation using  RC QP
    to configure the device steering.

Command utility - This module holds all of the FW commands that are
    required for SW steering to function.

Patch planning and files:
-------------------------
1) First patch, adds the support to Add flow steering actions to fs_cmd
shim layer.

2) Next 12 patch will add a file per each Software steering
functionality/module as described above. (See patches with title: DR, *)

3) Add CONFIG_MLX5_SW_STEERING for software steering support and enable
build with the new files

4) Next two patches will add the support for software steering in mlx5
steering shim layer
net/mlx5: Add API to set the namespace steering mode
net/mlx5: Add direct rule fs_cmd implementation

5) Last two patches will add the new devlink parameter to select mlx5
steering mode, will be valid only for switchdev mode for now.
Two modes are supported:
    1. DMFS - Device managed flow steering
    2. SMFS - Software/Driver managed flow steering.

    In the DMFS mode, the HW steering entities are created through the
    FW. In the SMFS mode this entities are created though the driver
    directly.

    The driver will use the devlink steering mode only if the steering
    domain supports it, for now SMFS will manages only the switchdev
    eswitch steering domain.

    User command examples:
    - Set SMFS flow steering mode::

        $ devlink dev param set pci/0000:06:00.0 name flow_steering_mode value "smfs" cmode runtime

    - Read device flow steering mode::

        $ devlink dev param show pci/0000:06:00.0 name flow_steering_mode
          pci/0000:06:00.0:
          name flow_steering_mode type driver-specific
          values:
             cmode runtime value smfs

----------------------------------------------------------------
Alex Vesker (13):
      net/mlx5: DR, Add the internal direct rule types definitions
      net/mlx5: DR, Add direct rule command utilities
      net/mlx5: DR, ICM pool memory allocator
      net/mlx5: DR, Expose an internal API to issue RDMA operations
      net/mlx5: DR, Add Steering entry (STE) utilities
      net/mlx5: DR, Expose steering domain functionality
      net/mlx5: DR, Expose steering table functionality
      net/mlx5: DR, Expose steering matcher functionality
      net/mlx5: DR, Expose steering action functionality
      net/mlx5: DR, Expose steering rule functionality
      net/mlx5: DR, Add required FW steering functionality
      net/mlx5: DR, Expose APIs for direct rule managing
      net/mlx5: DR, Add CONFIG_MLX5_SW_STEERING for software steering support

Maor Gottlieb (5):
      net/mlx5: Add flow steering actions to fs_cmd shim layer
      net/mlx5: Add direct rule fs_cmd implementation
      net/mlx5: Add API to set the namespace steering mode
      net/mlx5: Add support to use SMFS in switchdev mode
      net/mlx5: Add devlink flow_steering_mode parameter

 .../networking/device_drivers/mellanox/mlx5.rst    |   33 +
 drivers/infiniband/hw/mlx5/flow.c                  |   18 +-
 drivers/infiniband/hw/mlx5/main.c                  |    7 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h               |    5 +-
 drivers/net/ethernet/mellanox/mlx5/core/Kconfig    |    7 +
 drivers/net/ethernet/mellanox/mlx5/core/Makefile   |    7 +
 drivers/net/ethernet/mellanox/mlx5/core/devlink.c  |  112 +-
 .../net/ethernet/mellanox/mlx5/core/en/tc_tun.c    |   27 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_rep.h   |    2 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_tc.c    |   46 +-
 drivers/net/ethernet/mellanox/mlx5/core/eswitch.h  |    7 +-
 .../ethernet/mellanox/mlx5/core/eswitch_offloads.c |   87 +-
 drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c   |  116 +-
 drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.h   |   25 +
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.c  |  160 +-
 drivers/net/ethernet/mellanox/mlx5/core/fs_core.h  |   39 +-
 .../ethernet/mellanox/mlx5/core/steering/Makefile  |    2 +
 .../mellanox/mlx5/core/steering/dr_action.c        | 1588 ++++++++++++++
 .../ethernet/mellanox/mlx5/core/steering/dr_cmd.c  |  480 ++++
 .../mellanox/mlx5/core/steering/dr_crc32.c         |   98 +
 .../mellanox/mlx5/core/steering/dr_domain.c        |  395 ++++
 .../ethernet/mellanox/mlx5/core/steering/dr_fw.c   |   93 +
 .../mellanox/mlx5/core/steering/dr_icm_pool.c      |  570 +++++
 .../mellanox/mlx5/core/steering/dr_matcher.c       |  770 +++++++
 .../ethernet/mellanox/mlx5/core/steering/dr_rule.c | 1243 +++++++++++
 .../ethernet/mellanox/mlx5/core/steering/dr_send.c |  976 +++++++++
 .../ethernet/mellanox/mlx5/core/steering/dr_ste.c  | 2308 ++++++++++++++++++++
 .../mellanox/mlx5/core/steering/dr_table.c         |  294 +++
 .../mellanox/mlx5/core/steering/dr_types.h         | 1060 +++++++++
 .../ethernet/mellanox/mlx5/core/steering/fs_dr.c   |  600 +++++
 .../ethernet/mellanox/mlx5/core/steering/fs_dr.h   |   60 +
 .../mellanox/mlx5/core/steering/mlx5_ifc_dr.h      |  604 +++++
 .../ethernet/mellanox/mlx5/core/steering/mlx5dr.h  |  212 ++
 include/linux/mlx5/fs.h                            |   33 +-
 34 files changed, 11964 insertions(+), 120 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/Makefile
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_action.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_cmd.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_crc32.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_domain.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_fw.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_icm_pool.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_matcher.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_rule.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_table.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/dr_types.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/fs_dr.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5_ifc_dr.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/steering/mlx5dr.h

Powered by blists - more mailing lists