Message-Id: <20181221070647.12219-1-saeedm@mellanox.com>
Date:   Thu, 20 Dec 2018 23:06:38 -0800
From:   Saeed Mahameed <saeedm@...lanox.com>
To:     "David S. Miller" <davem@...emloft.net>
Cc:     netdev@...r.kernel.org, Tariq Toukan <tariqt@...lanox.com>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Tal Gilboa <talgi@...lanox.com>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Pawel Staszewski <pstaszewski@...are.pl>,
        Saeed Mahameed <saeedm@...lanox.com>
Subject: [pull request][net-next 0/9] Mellanox, mlx5 XDP TX 100 Million packet per second

Hi Dave,

This great series from Tariq is a very big leap and important milestone
for XDP. The XDP TX 100Mpps milestone is reached as Tariq adds the
required hardware support to overcome the pcie and cpu limits with the
ConnectX-5 Multi packet TX WQE feature (MPWQE), it is only a mater of time
to further improve this number as we still have some more tricks up our
sleeves.

For more information please see tag log below.

Please pull and let me know if there is any problem.

Thanks,
Saeed.

---
The following changes since commit a02e8964eaf9271a8a5fcc0c55bd13f933bafc56:

  virtio-net: ethtool configurable LRO (2018-12-20 19:14:15 -0800)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-XDP-100Mpps

for you to fetch changes up to 6277053afa34253eb8d8d147388e4426e615bae4:

  net/mlx5e: XDP, Add user control for XDP TX MPWQE feature (2018-12-20 22:54:20 -0800)

----------------------------------------------------------------
mlx5-XDP-100Mpps

This series from Tariq, mainly adds the support of mlx5 Multi Packet WQE
(TX descriptor) - ConnectX-5 and above - for XDP TX, which allows us to
overcome the 70Mpps PCIe bottleneck of conventional TX queues (single TX
descriptor per packet), and achieve the 100Mpps milestone with the MPWQE
approach.

In the first five patches, Tariq did minor improvements to mlx5 tx path,
for better debug-ability and code structuring.

Next two patches lay down the foundation for MPWQE implementation to store
the in-flight XDP TX information for multiple packets of one descriptor
(WQE).

Next: Support Enhanced Multi-Packet TX WQE for XDP

In this patch we add support for the HW feature, which is supported
starting from ConnectX-5.

Performance:
Tested packet rate for UDP 64Byte multi-stream over ConnectX-5 NICs.
CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

XDP_TX:
We see a huge gain on single port ConnectX-5, and reach the 100 Mpps
milestone.
* Single-port HCA:
	Before:   70 Mpps
	After:   100 Mpps (+42.8%)

* Dual-port HCA:
	Before: 51.7 Mpps
	After:  57.3 Mpps (+10.8%)

* In both cases we tested traffic on one port and for now On Dual-port
  HCAs we see only a small gain, we are working to overcome this
  bottleneck, but for the moment only with experimental firmware on dual
  port HCAs we can reach the wanted numbers as seen on Single-port HCAs.

XDP_REDIRECT:
Redirect from (A) ConnectX-5 to (B) ConnectX-5.
Due to a setup limitation, (A) and (B) are on different NUMA nodes,
so absolute performance numbers are not optimal.
- Note:
  Below is the transmit rate of (B), not the redirect rate of (A)
  which is in some cases higher.

* (B) is single-port:
	Before:   77 Mpps
	After:    90 Mpps (+16.8%)

* (B) is dual-port:
	Before:  61 Mpps
	After:   72 Mpps (+18%)

Last patch adds a knob in mlx5 ethtool private flag to turn on/off
XDP TX MPWQE.

-Saeed.

----------------------------------------------------------------
Tariq Toukan (9):
      net/mlx5e: TX, Print opcode in error CQE warning
      net/mlx5e: XDP, Precede XDP-related operations in RQ poll by a loaded program check
      net/mlx5e: XDP, Change the XDP SQ redirect indication
      net/mlx5e: XDP, Warn upon polling an error CQE
      net/mlx5e: XDP, Replace boolean doorbell indication with segment pointer
      net/mlx5e: XDP, Maintain a FIFO structure for xdp_info instances
      net/mlx5e: XDP, Add array for WQE info descriptors
      net/mlx5e: XDP, Support Enhanced Multi-Packet TX WQE
      net/mlx5e: XDP, Add user control for XDP TX MPWQE feature

 drivers/net/ethernet/mellanox/mlx5/core/en.h       |  35 +++-
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c   | 222 ++++++++++++++++-----
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h   |  55 ++++-
 .../net/ethernet/mellanox/mlx5/core/en_ethtool.c   |  28 +++
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c  |  95 ++++++---
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c    |  12 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_tx.c    |   7 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c  |  13 +-
 include/linux/mlx5/device.h                        |   1 +
 9 files changed, 359 insertions(+), 109 deletions(-)

Powered by blists - more mailing lists