netdev - Problem of rate limit with VXLAN offload Mellanox ConnectX-5 NIC

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <9554367f-5ed5-42f1-af77-c72534985f04@www.fastmail.com>
Date:   Thu, 28 Feb 2019 03:55:45 -0500
From:   "Jingrong Chen" <jchendi@....ust.hk>
To:     netdev@...r.kernel.org, nic-support@...lanox.com,
        jzhangcs@...nect.ust.hk, shuaa@....ust.hk
Subject: Problem of rate limit with VXLAN offload Mellanox ConnectX-5 NIC

Hi all,

We ran into a performance issue when configuring the ConnectX-5 NICs to do both tc rate limiting and VXLAN encapsulation/decapsulation simultaneously.

Our test setting is as follows: 
We have two servers, each installed one ConnectX-5 NIC. Each server runs one VM. We use ib_write_bw to let one VM send RDMA traffic to the other VM. The traffic priority is configured to be 3.
1) We configure tc rate limiting using command "mlnx_qos -i rdma0 -r 0,0,0,X,0,0,0,0" (i.e., limit the traffic rate to be X Gbps).
2) We enable SR-IOV at the NIC and follow the ASAP2 Hardware Offloading for vSwitches User Manual to configure vxlan encapsulation/decapsulation for the RDMA traffic.

Our test result is as below:
Rate limit (Gbps)     Throughput (Gbps)
X = 40                       19.09  (throughput only achieved half of the configured rate)
X = 60                       28.66
X = 80                       38.25
X = 100                     47.77
unlimited                   87.68

* Note that when only enabling tc rate limiting, there is no such throughput downgrade issue.

It would be of great help if anyone could help us to resolve this problem. Thanks.

FYI, our machines run Ubuntu 18.04, the kernel version is 4.15.0, CX5 firmware is 16.24.1000, and the driver is MLNX_OFED_LINUX-4.5-1.0.1.0.

Regards,
Jingrong