netdev - Re: XDP Performance Regression in recent kernel versions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+h3auMq5vnoyRLvJainG-AFA6f=ivRmu6RjKU4cBv_go975tw@mail.gmail.com>
Date: Fri, 21 Jun 2024 14:35:06 +0200
From: Samuel Dobron <sdobron@...hat.com>
To: Daniel Borkmann <daniel@...earbox.net>, hawk@...nel.org
Cc: Toke Høiland-Jørgensen <toke@...hat.com>, 
	Sebastiano Miano <mianosebastiano@...il.com>, bpf@...r.kernel.org, netdev@...r.kernel.org, 
	saeedm@...dia.com, tariqt@...dia.com, edumazet@...gle.com, kuba@...nel.org, 
	pabeni@...hat.com
Subject: Re: XDP Performance Regression in recent kernel versions

Hey all,

Yeah, we do tests for ELN kernels [1] on a regular basis. Since
~January of this year.

As already mentioned, mlx5 is the only driver affected by this regression.
Unfortunately, I think Jesper is actually hitting 2 regressions we noticed,
the one already mentioned by Toke, another one [0] has been reported
in early February.
Btw. issue mentioned by Toke has been moved to Jira, see [5].

Not sure all of you are able to see the content of [0], Jira says it's
RH-confidental.
So, I am not sure how much I can share without being fired :D. Anyway,
affected kernels have been released a while ago, so anyone can find it
on its own.
Basically, we detected 5% regression on XDP_DROP+mlx5 (currently, we
don't have data for any other XDP mode) in kernel-5.14 compared to
previous builds.

>From tests history, I can see (most likely) the same improvement
on 6.10rc2 (from 15Mpps to 17-18Mpps), so I'd say 20% drop has been
(partially) fixed?

For earlier 6.10. kernels we don't have data due to [3] (there is regression on
XDP_DROP as well, but I believe it's turbo-boost issue, as I mentioned
in issue).
So if you want to run tests on 6.10. please see [3].

Summary XDP_DROP+mlx5@25G:
kernel       pps
<5.14        20.5M        baseline
>=5.14      19M           [0]
<6.4          19-20M      baseline for ELN kernels
>=6.4        15M           [4 and 5] (mentioned by Toke)
>=6.10      ???            [3]
>=6.10rc2 17M-18M

> It looks like this is known since March, was this ever reported to Nvidia back
> then? :/

Not sure if that's a question for me, I was told, filling an issue in
Bugzilla/Jira is where
our competences end. Who is supposed to report it to them?

> Given XDP is in the critical path for many in production, we should think about
> regular performance reporting for the different vendors for each released kernel,
> similar to here [0].

I think this might be the part of upstream kernel testing with LNST?
Maybe Jesper
knows more about that? Until then, I think, I can let you know about
new regressions we catch.

Thanks,
Sam.

[0] https://issues.redhat.com/browse/RHEL-24054
[1] https://koji.fedoraproject.org/koji/search?terms=kernel-%5Cd.*eln*&type=build&match=regexp
[2] https://koji.fedoraproject.org/koji/buildinfo?buildID=2469107
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2282969
[4] https://bugzilla.redhat.com/show_bug.cgi?id=2270408
[5] https://issues.redhat.com/browse/RHEL-24054