netdev - Re: Need of advice for XDP sockets on top of the interfaces behind a Linux bonding device

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <87r0hyzxbd.fsf@toke.dk>
Date: Tue, 30 Jan 2024 15:54:14 +0100
From: Toke Høiland-Jørgensen <toke@...nel.org>
To: Pavel Vazharov <pavel@...e.net>
Cc: Jakub Kicinski <kuba@...nel.org>, netdev@...r.kernel.org, Magnus
 Karlsson <magnus.karlsson@...il.com>
Subject: Re: Need of advice for XDP sockets on top of the interfaces behind
 a Linux bonding device

Pavel Vazharov <pavel@...e.net> writes:

> On Tue, Jan 30, 2024 at 4:32 PM Toke Høiland-Jørgensen <toke@...nel.org> wrote:
>>
>> Pavel Vazharov <pavel@...e.net> writes:
>>
>> >> On Sat, Jan 27, 2024 at 7:08 AM Pavel Vazharov <pavel@...e.net> wrote:
>> >>>
>> >>> On Sat, Jan 27, 2024 at 6:39 AM Jakub Kicinski <kuba@...nel.org> wrote:
>> >>> >
>> >>> > On Sat, 27 Jan 2024 05:58:55 +0200 Pavel Vazharov wrote:
>> >>> > > > Well, it will be up to your application to ensure that it is not. The
>> >>> > > > XDP program will run before the stack sees the LACP management traffic,
>> >>> > > > so you will have to take some measure to ensure that any such management
>> >>> > > > traffic gets routed to the stack instead of to the DPDK application. My
>> >>> > > > immediate guess would be that this is the cause of those warnings?
>> >>> > >
>> >>> > > Thank you for the response.
>> >>> > > I already checked the XDP program.
>> >>> > > It redirects particular pools of IPv4 (TCP or UDP) traffic to the application.
>> >>> > > Everything else is passed to the Linux kernel.
>> >>> > > However, I'll check it again. Just to be sure.
>> >>> >
>> >>> > What device driver are you using, if you don't mind sharing?
>> >>> > The pass thru code path may be much less well tested in AF_XDP
>> >>> > drivers.
>> >>> These are the kernel version and the drivers for the 3 ports in the
>> >>> above bonding.
>> >>> ~# uname -a
>> >>> Linux 6.3.2 #1 SMP Wed May 17 08:17:50 UTC 2023 x86_64 GNU/Linux
>> >>> ~# lspci -v | grep -A 16 -e 1b:00.0 -e 3b:00.0 -e 5e:00.0
>> >>> 1b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> >>> SFI/SFP+ Network Connection (rev 01)
>> >>>        ...
>> >>>         Kernel driver in use: ixgbe
>> >>> --
>> >>> 3b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> >>> SFI/SFP+ Network Connection (rev 01)
>> >>>         ...
>> >>>         Kernel driver in use: ixgbe
>> >>> --
>> >>> 5e:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit
>> >>> SFI/SFP+ Network Connection (rev 01)
>> >>>         ...
>> >>>         Kernel driver in use: ixgbe
>> >>>
>> >>> I think they should be well supported, right?
>> >>> So far, it seems that the present usage scenario should work and the
>> >>> problem is somewhere in my code.
>> >>> I'll double check it again and try to simplify everything in order to
>> >>> pinpoint the problem.
>> > I've managed to pinpoint that forcing the copying of the packets
>> > between the kernel and the user space
>> > (XDP_COPY) fixes the issue with the malformed LACPDUs and the not
>> > working bonding.
>>
>> (+Magnus)
>>
>> Right, okay, that seems to suggest a bug in the internal kernel copying
>> that happens on XDP_PASS in zero-copy mode. Which would be a driver bug;
>> any chance you could test with a different driver and see if the same
>> issue appears there?
>>
>> -Toke
> No, sorry.
> We have only servers with Intel 82599ES with ixgbe drivers.
> And one lab machine with Intel 82540EM with igb driver but we can't
> set up bonding there
> and the problem is not reproducible there.

Right, okay. Another thing that may be of some use is to try to capture
the packets on the physical devices using tcpdump. That should (I think)
show you the LACDPU packets as they come in, before they hit the bonding
device, but after they are copied from the XDP frame. If it's a packet
corruption issue, that should be visible in the captured packet; you can
compare with an xdpdump capture to see if there are any differences...

-Toke