netdev - Re: ixgbe driver stops sending normal data when using xsk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aH4HNquLzjagCLeX@soc-5CG4396X81.clients.intel.com>
Date: Mon, 21 Jul 2025 11:24:06 +0200
From: Larysa Zaremba <larysa.zaremba@...el.com>
To: Jason Xing <kerneljasonxing@...il.com>
CC: Jakub Kicinski <kuba@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>,
	"Nguyen, Anthony L" <anthony.l.nguyen@...el.com>,
	<przemyslaw.kitszel@...el.com>, Maciej Fijalkowski
	<maciej.fijalkowski@...el.com>, <intel-wired-lan@...ts.osuosl.org>, netdev
	<netdev@...r.kernel.org>
Subject: Re: ixgbe driver stops sending normal data when using xsk

On Mon, Jul 21, 2025 at 05:03:07PM +0800, Jason Xing wrote:
> On Mon, Jul 21, 2025 at 3:25 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
> >
> > On Sat, Jul 19, 2025 at 01:26:18PM +0800, Jason Xing wrote:
> > > On Fri, Jul 18, 2025 at 6:27 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
> > > >
> > > > On Wed, Jul 16, 2025 at 11:41:42AM +0800, Jason Xing wrote:
> > > > > Hi all,
> > > > >
> > > > > I'm currently faced with one tough issue caused by zero copy mode in
> > > > > xsk with ixgbe driver loaded. The case is that if we use xdpsock to
> > > > > send descs, nearly at the same time normal packets from other tx
> > > > > queues cannot be transmitted/completed at all.
> > > > >
> > > > > Here is how I try:
> > > > > 1. run iperf or ping to see if the transmission is successful.
> > > > > 2. then run "timeout 5 ./xdpsock -i enp2s0f0 -t  -z -s 64"
> > > > >
> > > > > You will obviously find the whole machine loses connection. It can
> > > > > only recover as soon as the xdpsock is stopped due to timeout.
> > > > >
> > > > > I tried a lot and then traced down to this line in ixgbe driver:
> > > > > ixgbe_clean_tx_irq()
> > > > >     -> if (!(eop_desc->wb.status & cpu_to_le32(IXGBE_TXD_STAT_DD)))
> > > > >             break;
> > > > > The above line always 'breaks' the sending process.
> > > > >
> > > > > I also managed to make the external ixgbe 6.15 work and it turned out
> > > > > to be the same issue as before.
> > > > >
> > > > > I have no idea on how to analyze further in this driver. Could someone
> > > > > point out a direction that I can take? Is it a known issue?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > >
> > > > I was able to reproduce the described behaviour, xdpsock does break the IP
> > > > communication. However, in my case this was not because of ixgbe not being able
> > > > to send, but because of queue 0 RX packets being dropped, which is the indended
> > > > outcome in xdpsock, even in Tx only mode.
> > >
> > > Thanks for your feedback. It would be great if you could elaborate
> > > more on this. How did you spot that it's queue 0 that causes the
> > > problem?
> >
> > If you do not specify -q parameter, xdpsock loads on the queue pair 0.
> >
> > > Why is xdpsock breaking IP communication intended?
> >
> > Because when a packet arrives on the AF_XDP-managed queue (0 in this case), the
> > default xdpsock XDP program provided by libxdp returns XDP_REDIRECT even in
> > tx-only mode, XDP_PASS for all other queues (1-39). XDP_REDIRECT results in a
> > packet leaving the kernel network stack, it is now managed by the AF_XDP
> > userspace program. I think it is possible to modify libxdp to return XDP_PASS
> > when the socket is tx-only.
> >
> > >
> > > When you try i40e, you will find the connection behaves normally. Ping
> > > can work as usual. As I depicted before, with ixgbe driver, ping even
> > > doesn't work at all.
> >
> > I think this is due to RSS configuration, ping packets on i40e go to another
> > queue.
> 
> Thanks so much for your detailed explanation.
> 
> But, I still doubt it's not the reason why xdpsock breaks with ixgbe
> driver loaded because I tried the following commands:
> 1. ping <ip>
> 2. timeout 3 ./xdpsock -i enp2s0f0np0 -t -q 11  -z -s 64
> Whatever the queue value I adjusted, ping always fails with
> "Destination Host Unreachable" warning.

First I would make sure, if the problem is from Rx or Tx. This is hard to do 
with IP-level applications, because they fail if any of those does not work.

Please, try sending MAC packet with scapy using a following command:
sendp(Ether(src="<src MAC>", dst="<dst MAC>")/IP(src="<any IP>", dst="<another IP>")/UDP(sport=2000, dport=9091)/Raw(load="xdp"), iface="<src ifname>")

First, send this from host to link partner while running `tcpdump -nn -e -p -i 
<ifname>` on link partner.

Then send this from LK to host while running tcpdump on host.

In both cases, note whether the packet shows up in tcpdump with and without 
xdpsock loaded on the host.

> 
> Thanks,
> Jason