netdev - Re: ixgbe driver stops sending normal data when using xsk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAL+tcoANWC5h2FdRkB470qmWYk9-f-SVbRmwiNqfXRXJn8JZBQ@mail.gmail.com>
Date: Mon, 21 Jul 2025 17:56:37 +0800
From: Jason Xing <kerneljasonxing@...il.com>
To: Larysa Zaremba <larysa.zaremba@...el.com>
Cc: Jakub Kicinski <kuba@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, 
	"Nguyen, Anthony L" <anthony.l.nguyen@...el.com>, przemyslaw.kitszel@...el.com, 
	Maciej Fijalkowski <maciej.fijalkowski@...el.com>, intel-wired-lan@...ts.osuosl.org, 
	netdev <netdev@...r.kernel.org>
Subject: Re: ixgbe driver stops sending normal data when using xsk

On Mon, Jul 21, 2025 at 5:25 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
>
> On Mon, Jul 21, 2025 at 05:03:07PM +0800, Jason Xing wrote:
> > On Mon, Jul 21, 2025 at 3:25 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
> > >
> > > On Sat, Jul 19, 2025 at 01:26:18PM +0800, Jason Xing wrote:
> > > > On Fri, Jul 18, 2025 at 6:27 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
> > > > >
> > > > > On Wed, Jul 16, 2025 at 11:41:42AM +0800, Jason Xing wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I'm currently faced with one tough issue caused by zero copy mode in
> > > > > > xsk with ixgbe driver loaded. The case is that if we use xdpsock to
> > > > > > send descs, nearly at the same time normal packets from other tx
> > > > > > queues cannot be transmitted/completed at all.
> > > > > >
> > > > > > Here is how I try:
> > > > > > 1. run iperf or ping to see if the transmission is successful.
> > > > > > 2. then run "timeout 5 ./xdpsock -i enp2s0f0 -t  -z -s 64"
> > > > > >
> > > > > > You will obviously find the whole machine loses connection. It can
> > > > > > only recover as soon as the xdpsock is stopped due to timeout.
> > > > > >
> > > > > > I tried a lot and then traced down to this line in ixgbe driver:
> > > > > > ixgbe_clean_tx_irq()
> > > > > >     -> if (!(eop_desc->wb.status & cpu_to_le32(IXGBE_TXD_STAT_DD)))
> > > > > >             break;
> > > > > > The above line always 'breaks' the sending process.
> > > > > >
> > > > > > I also managed to make the external ixgbe 6.15 work and it turned out
> > > > > > to be the same issue as before.
> > > > > >
> > > > > > I have no idea on how to analyze further in this driver. Could someone
> > > > > > point out a direction that I can take? Is it a known issue?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > >
> > > > > I was able to reproduce the described behaviour, xdpsock does break the IP
> > > > > communication. However, in my case this was not because of ixgbe not being able
> > > > > to send, but because of queue 0 RX packets being dropped, which is the indended
> > > > > outcome in xdpsock, even in Tx only mode.
> > > >
> > > > Thanks for your feedback. It would be great if you could elaborate
> > > > more on this. How did you spot that it's queue 0 that causes the
> > > > problem?
> > >
> > > If you do not specify -q parameter, xdpsock loads on the queue pair 0.
> > >
> > > > Why is xdpsock breaking IP communication intended?
> > >
> > > Because when a packet arrives on the AF_XDP-managed queue (0 in this case), the
> > > default xdpsock XDP program provided by libxdp returns XDP_REDIRECT even in
> > > tx-only mode, XDP_PASS for all other queues (1-39). XDP_REDIRECT results in a
> > > packet leaving the kernel network stack, it is now managed by the AF_XDP
> > > userspace program. I think it is possible to modify libxdp to return XDP_PASS
> > > when the socket is tx-only.
> > >
> > > >
> > > > When you try i40e, you will find the connection behaves normally. Ping
> > > > can work as usual. As I depicted before, with ixgbe driver, ping even
> > > > doesn't work at all.
> > >
> > > I think this is due to RSS configuration, ping packets on i40e go to another
> > > queue.
> >
> > Thanks so much for your detailed explanation.
> >
> > But, I still doubt it's not the reason why xdpsock breaks with ixgbe
> > driver loaded because I tried the following commands:
> > 1. ping <ip>
> > 2. timeout 3 ./xdpsock -i enp2s0f0np0 -t -q 11  -z -s 64
> > Whatever the queue value I adjusted, ping always fails with
> > "Destination Host Unreachable" warning.
>
> First I would make sure, if the problem is from Rx or Tx. This is hard to do

I ran tcpdump xxx & on the other machine (receiver) to see if there
were any incoming icmp packets first. It turned out to be no incoming
data at all. So I presume the problem happens on the tx side.

> with IP-level applications, because they fail if any of those does not work.
>
> Please, try sending MAC packet with scapy using a following command:
> sendp(Ether(src="<src MAC>", dst="<dst MAC>")/IP(src="<any IP>", dst="<another IP>")/UDP(sport=2000, dport=9091)/Raw(load="xdp"), iface="<src ifname>")
>
> First, send this from host to link partner while running `tcpdump -nn -e -p -i
> <ifname>` on link partner.
>
> Then send this from LK to host while running tcpdump on host.
>
> In both cases, note whether the packet shows up in tcpdump with and without
> xdpsock loaded on the host.

Thanks for your guidance. But it probably takes me more time than
expected due to the whole connection being totally broken while
xdpsock is running :(

I ran tcpdump xxx > log & while running xdpsock. Later, I found that
there are as many packets as w/o xdpscok.

Thanks,
Jason