netdev - Re: XDP Performance Regression in recent kernel versions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+h3auNx4jTALyhYAm9w6xaObnTvyCAMp7pNTOym5jcX5rJz=A@mail.gmail.com>
Date: Thu, 25 Jul 2024 14:27:43 +0200
From: Samuel Dobron <sdobron@...hat.com>
To: Toke Høiland-Jørgensen <toke@...hat.com>
Cc: Carolina Jubran <cjubran@...dia.com>, Dragos Tatulea <dtatulea@...dia.com>, 
	Tariq Toukan <tariqt@...dia.com>, "daniel@...earbox.net" <daniel@...earbox.net>, 
	"hawk@...nel.org" <hawk@...nel.org>, "mianosebastiano@...il.com" <mianosebastiano@...il.com>, 
	"pabeni@...hat.com" <pabeni@...hat.com>, "netdev@...r.kernel.org" <netdev@...r.kernel.org>, 
	"edumazet@...gle.com" <edumazet@...gle.com>, Saeed Mahameed <saeedm@...dia.com>, 
	"bpf@...r.kernel.org" <bpf@...r.kernel.org>, "kuba@...nel.org" <kuba@...nel.org>
Subject: Re: XDP Performance Regression in recent kernel versions

Confirming that this is just mlx5 issue, intel is fine.

I just did a quick test with disabled[0] Spectre v2 mitigations.
The performance remains the same, no difference at all.

Sam.

[0]:
$ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Vulnerable;
BHI: Vulnerable

On Wed, Jul 24, 2024 at 5:48 PM Toke Høiland-Jørgensen <toke@...hat.com> wrote:
>
> Carolina Jubran <cjubran@...dia.com> writes:
>
> > On 22/07/2024 12:26, Dragos Tatulea wrote:
> >> On Sun, 2024-06-30 at 14:43 +0300, Tariq Toukan wrote:
> >>>
> >>> On 21/06/2024 15:35, Samuel Dobron wrote:
> >>>> Hey all,
> >>>>
> >>>> Yeah, we do tests for ELN kernels [1] on a regular basis. Since
> >>>> ~January of this year.
> >>>>
> >>>> As already mentioned, mlx5 is the only driver affected by this regression.
> >>>> Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
> >>>> the one already mentioned by Toke, another one [0] has been reported
> >>>> in early February.
> >>>> Btw. issue mentioned by Toke has been moved to Jira, see [5].
> >>>>
> >>>> Not sure all of you are able to see the content of [0], Jira says it's
> >>>> RH-confidental.
> >>>> So, I am not sure how much I can share without being fired :D. Anyway,
> >>>> affected kernels have been released a while ago, so anyone can find it
> >>>> on its own.
> >>>> Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
> >>>> don't have data for any other XDP mode) in kernel-5.14 compared to
> >>>> previous builds.
> >>>>
> >>>>   From tests history, I can see (most likely) the same improvement
> >>>> on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
> >>>> (partially) fixed?
> >>>>
> >>>> For earlier 6.10. kernels we don't have data due to [3] (there is regression on
> >>>> XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
> >>>> in issue).
> >>>> So if you want to run tests on 6.10. please see [3].
> >>>>
> >>>> Summary XDP_DROP+mlx5@25G:
> >>>> kernel       pps
> >>>> <5.14        20.5M        baseline
> >>>>> =5.14      19M           [0]
> >>>> <6.4          19-20M      baseline for ELN kernels
> >>>>> =6.4        15M           [4 and 5] (mentioned by Toke)
> >>>
> >>> + @Dragos
> >>>
> >>> That's about when we added several changes to the RX datapath.
> >>> Most relevant are:
> >>> - Fully removing the in-driver RX page-cache.
> >>> - Refactoring to support XDP multi-buffer.
> >>>
> >>> We tested XDP performance before submission, I don't recall we noticed
> >>> such a degradation.
> >>
> >> Adding Carolina to post her analysis on this.
> >
> > Hey everyone,
> >
> > After investigating the issue, it seems the performance degradation is
> > linked to the commit "x86/bugs: Report Intel retbleed vulnerability"
> > (6ad0ad2bf8a67).
>
> Hmm, that commit is from June 2022, and according to Samuel's tests,
> this issue was introduced sometime between commits b6dad5178cea and
> 40f71e7cd3c6 (both of which are dated in June 2023). Besides, if it was
> a retbleed mitigation issue, that would affect other drivers as well,
> no? Our testing only shows this regression on mlx5, not on the intel
> drivers.
>
>
> >>> I'll check with Dragos as he probably has these reports.
> >>>
> >> We only noticed a 6% degradation for XDP_XDROP.
> >>
> >> https://lore.kernel.org/netdev/b6fcfa8b-c2b3-8a92-fb6e-0760d5f6f5ff@redhat.com/T/
>
> That message mentions that "This will be handled in a different patch
> series by adding support for multi-packet per page." - did that ever go
> in?
>
> -Toke
>