netdev - Re: XDP Performance Regression in recent kernel versions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Thu, 20 Jun 2024 11:52:22 +0200
From: Daniel Borkmann <daniel@...earbox.net>
To: Toke Høiland-Jørgensen <toke@...hat.com>,
 Jesper Dangaard Brouer <hawk@...nel.org>,
 Sebastiano Miano <mianosebastiano@...il.com>, bpf@...r.kernel.org,
 netdev@...r.kernel.org
Cc: saeedm@...dia.com, tariqt@...dia.com, edumazet@...gle.com,
 kuba@...nel.org, pabeni@...hat.com, Samuel Dobron <sdobron@...hat.com>,
 netdev@...r.kernel.org
Subject: Re: XDP Performance Regression in recent kernel versions

On 6/19/24 9:17 PM, Toke Høiland-Jørgensen wrote:
> Jesper Dangaard Brouer <hawk@...nel.org> writes:
>> On 18/06/2024 17.28, Sebastiano Miano wrote:
>>> I have been conducting some basic experiments with XDP and have
>>> observed a significant performance regression in recent kernel
>>> versions compared to v5.15.
>>>
>>> My setup is the following:
>>> - Hardware: Two machines connected back-to-back with 100G Mellanox
>>> ConnectX-6 Dx.
>>> - DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
>>> - Software: xdp-bench program from [1] running on the DUT in both DROP
>>> and TX modes.
>>> - Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
>>> flow at ~130Mpps.
>>> - Tests: Single core, HT disabled
>>>
>>> Results:
>>>
>>> Kernel version |-------| XDP_DROP |--------|   XDP_TX  |
>>> 5.15                      30Mpps               16.1Mpps
>>> 6.2                       21.3Mpps             14.1Mpps
>>> 6.5                       19.9Mpps              8.6Mpps
>>> bpf-next (6.10-rc2)       22.1Mpps              9.2Mpps
>>>
>>
>> Around when I left Red Hat there were a project with [LNST] that used
>> xdp-bench for tracking and finding regressions like this.
>>
>> Perhaps Toke can enlighten us, if that project have caught similar
>> regressions?
>>
>> [LNST] https://github.com/LNST-project/lnst
> 
> Yes, actually, we have! Here's the bugzilla for it:
> https://bugzilla.redhat.com/show_bug.cgi?id=2270408

 > We compared performance of ELN and RHEL9 candidate kernels and noticed significant
 > drop in XDP drop [1] on mlx5 (25G).
 >
 > On any rhel9 candidate kernel we are able to drop 19-20M pkts/sec but on an ELN
 > kernels, we are reaching just 15M pkts/sec (CPU utillization remains the same -
 > around 100%).
 >
 > We don't see such regression on ixgbe or i40e.

It looks like this is known since March, was this ever reported to Nvidia back
then? :/

Given XDP is in the critical path for many in production, we should think about
regular performance reporting for the different vendors for each released kernel,
similar to here [0].

Out of curiosity, @Saeed: Is Nvidia internally regularly assessing XDP perf for mlx5
as part of QA? (Probably not, but I thought I'd ask.)

Thanks,
Daniel

   [0] http://core.dpdk.org/perf-reports/