netdev - Re: Need of advice for XDP sockets on top of the interfaces behind a Linux bonding device

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJ8uoz0bZjeJKJbCOfaogJZg2o5zoWNf3XjAF-4ZubpxDoQhcg@mail.gmail.com>
Date: Mon, 5 Feb 2024 08:07:02 +0100
From: Magnus Karlsson <magnus.karlsson@...il.com>
To: Toke Høiland-Jørgensen <toke@...nel.org>, 
	Pavel Vazharov <pavel@...e.net>
Cc: Jakub Kicinski <kuba@...nel.org>, netdev@...r.kernel.org, 
	"Fijalkowski, Maciej" <maciej.fijalkowski@...el.com>
Subject: Re: Need of advice for XDP sockets on top of the interfaces behind a
 Linux bonding device

On Tue, 30 Jan 2024 at 15:54, Toke Høiland-Jørgensen <toke@...nel.org> wrote:
>
> Pavel Vazharov <pavel@...e.net> writes:
>
> > On Tue, Jan 30, 2024 at 4:32 PM Toke Høiland-Jørgensen <toke@...nel.org> wrote:
> >>
> >> Pavel Vazharov <pavel@...e.net> writes:
> >>
> >> >> On Sat, Jan 27, 2024 at 7:08 AM Pavel Vazharov <pavel@...e.net> wrote:
> >> >>>
> >> >>> On Sat, Jan 27, 2024 at 6:39 AM Jakub Kicinski <kuba@...nel.org> wrote:
> >> >>> >
> >> >>> > On Sat, 27 Jan 2024 05:58:55 +0200 Pavel Vazharov wrote:
> >> >>> > > > Well, it will be up to your application to ensure that it is not. The
> >> >>> > > > XDP program will run before the stack sees the LACP management traffic,
> >> >>> > > > so you will have to take some measure to ensure that any such management
> >> >>> > > > traffic gets routed to the stack instead of to the DPDK application. My
> >> >>> > > > immediate guess would be that this is the cause of those warnings?
> >> >>> > >
> >> >>> > > Thank you for the response.
> >> >>> > > I already checked the XDP program.
> >> >>> > > It redirects particular pools of IPv4 (TCP or UDP) traffic to the application.
> >> >>> > > Everything else is passed to the Linux kernel.
> >> >>> > > However, I'll check it again. Just to be sure.
> >> >>> >
> >> >>> > What device driver are you using, if you don't mind sharing?
> >> >>> > The pass thru code path may be much less well tested in AF_XDP
> >> >>> > drivers.
> >> >>> These are the kernel version and the drivers for the 3 ports in the
> >> >>> above bonding.
> >> >>> ~# uname -a
> >> >>> Linux 6.3.2 #1 SMP Wed May 17 08:17:50 UTC 2023 x86_64 GNU/Linux
> >> >>> ~# lspci -v | grep -A 16 -e 1b:00.0 -e 3b:00.0 -e 5e:00.0
> >> >>> 1b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> >> >>> SFI/SFP+ Network Connection (rev 01)
> >> >>>        ...
> >> >>>         Kernel driver in use: ixgbe
> >> >>> --
> >> >>> 3b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> >> >>> SFI/SFP+ Network Connection (rev 01)
> >> >>>         ...
> >> >>>         Kernel driver in use: ixgbe
> >> >>> --
> >> >>> 5e:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
> >> >>> SFI/SFP+ Network Connection (rev 01)
> >> >>>         ...
> >> >>>         Kernel driver in use: ixgbe
> >> >>>
> >> >>> I think they should be well supported, right?
> >> >>> So far, it seems that the present usage scenario should work and the
> >> >>> problem is somewhere in my code.
> >> >>> I'll double check it again and try to simplify everything in order to
> >> >>> pinpoint the problem.
> >> > I've managed to pinpoint that forcing the copying of the packets
> >> > between the kernel and the user space
> >> > (XDP_COPY) fixes the issue with the malformed LACPDUs and the not
> >> > working bonding.
> >>
> >> (+Magnus)
> >>
> >> Right, okay, that seems to suggest a bug in the internal kernel copying
> >> that happens on XDP_PASS in zero-copy mode. Which would be a driver bug;
> >> any chance you could test with a different driver and see if the same
> >> issue appears there?
> >>
> >> -Toke
> > No, sorry.
> > We have only servers with Intel 82599ES with ixgbe drivers.
> > And one lab machine with Intel 82540EM with igb driver but we can't
> > set up bonding there
> > and the problem is not reproducible there.
>
> Right, okay. Another thing that may be of some use is to try to capture
> the packets on the physical devices using tcpdump. That should (I think)
> show you the LACDPU packets as they come in, before they hit the bonding
> device, but after they are copied from the XDP frame. If it's a packet
> corruption issue, that should be visible in the captured packet; you can
> compare with an xdpdump capture to see if there are any differences...

Pavel,

Sounds like an issue with the driver in zero-copy mode as it works
fine in copy mode. Maciej and I will take a look at it.

> -Toke
>