Message-ID: <CAJEV1ijxNyPTwASJER1bcZzS9nMoZJqfR86nu_3jFFVXzZQ4NA@mail.gmail.com>
Date: Fri, 26 Jan 2024 17:54:59 +0200
From: Pavel Vazharov <pavel@...e.net>
To: netdev@...r.kernel.org
Subject: Need of advice for XDP sockets on top of the interfaces behind a
 Linux bonding device

Hi there,

We've a DPDK application which runs on top of XDP sockets, using the
DPDK AF_XDP driver. It was a pure DPDK application but lately it was
migrated to run on top of XDP sockets because we need to split the
traffic entering the machine between the DPDK application and other
"standard-Linux" applications running on the same machine.
The application seems to work OK when working on a single port.
However, running on the interfaces behind a bonding device causes the
remote device (switch) to start reporting: "The member of the LACP
mode Eth-Trunk interface received an abnormal LACPDU, which may be
caused by optical fiber misconnection" and the bonding stops working.
Note that the application needs to work with multiple queues and thus
the XDP sockets are not bound to the bonding device but to the
physical interfaces behind the bonding device. As far as I checked the
bonding device supports binding only a single XDP socket and makes it
unusable for our purposes.
In the concrete example, there are 3 physical ports in bonding and
each port is set up to have 16 Rx/Tx (combined) queues. The
application (the DPDK layer) opens an XDP socket for each queue of the
physical ports (Basically the DPDK layer creates 3 virtual af_xdp
devices and each one of them has Rx/Tx 16 queues where each queue is
actually an XDP socket). I've run the application in different
threading scenarios but each one of them exhibit the above problem:
- single thread - where all of the Rx/Tx on the queues is handled by a
single thread
- two threads - where the first thread handles Rx/Tx on (dev:0
queues:0-15) and (dev:1 queues:0-7) and the second thread handles
Rx/Tx on (dev:1 queues:8-15) and (dev:2 queues:0-15).
- three threads - where the first thread handles Rx/Tx on (dev:0
queues:0-15), the second thread handles Rx/Tx on (dev:1 queues:0-15),
the third thread handles Rx/Tx on (dev:2 queues:0-15).
I've tried with and without busy polling in the above threading
schemes and the problem was still there.

Related to the above, I've the following questions:
1. Is it possible to use multiple XDP sockets with a bonding device? I
mean, if we use the above example, will it be possible to open 16 XDP
sockets on top of the bonding device which has 3 ports and each have
16 Rx/Tx queues.
2. If point 1 is not possible then is the above scheme supposed to
work in general or is it not right to bind the XDP sockets to the
queues of the underlying physical ports?
3. If the above scheme is supposed to work then is the bonding logic
(LACP management traffic) affected by the access pattern of the XDP
sockets? I mean, the order of Rx/Tx operations on the XDP sockets or
something like that.

Any other advice on what I should check again or change or research is
greatly appreciated.

Regards,
Pavel.

Powered by blists - more mailing lists