netdev - Re: ixgbe driver stops sending normal data when using xsk

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAL+tcoAQF_Aom4dn--RQzowiUO1haNPw=_Rzw2C7MJRF_sSUOw@mail.gmail.com>
Date: Mon, 21 Jul 2025 17:03:07 +0800
From: Jason Xing <kerneljasonxing@...il.com>
To: Larysa Zaremba <larysa.zaremba@...el.com>
Cc: Jakub Kicinski <kuba@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, 
	"Nguyen, Anthony L" <anthony.l.nguyen@...el.com>, przemyslaw.kitszel@...el.com, 
	Maciej Fijalkowski <maciej.fijalkowski@...el.com>, intel-wired-lan@...ts.osuosl.org, 
	netdev <netdev@...r.kernel.org>
Subject: Re: ixgbe driver stops sending normal data when using xsk

On Mon, Jul 21, 2025 at 3:25 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
>
> On Sat, Jul 19, 2025 at 01:26:18PM +0800, Jason Xing wrote:
> > On Fri, Jul 18, 2025 at 6:27 PM Larysa Zaremba <larysa.zaremba@...el.com> wrote:
> > >
> > > On Wed, Jul 16, 2025 at 11:41:42AM +0800, Jason Xing wrote:
> > > > Hi all,
> > > >
> > > > I'm currently faced with one tough issue caused by zero copy mode in
> > > > xsk with ixgbe driver loaded. The case is that if we use xdpsock to
> > > > send descs, nearly at the same time normal packets from other tx
> > > > queues cannot be transmitted/completed at all.
> > > >
> > > > Here is how I try:
> > > > 1. run iperf or ping to see if the transmission is successful.
> > > > 2. then run "timeout 5 ./xdpsock -i enp2s0f0 -t  -z -s 64"
> > > >
> > > > You will obviously find the whole machine loses connection. It can
> > > > only recover as soon as the xdpsock is stopped due to timeout.
> > > >
> > > > I tried a lot and then traced down to this line in ixgbe driver:
> > > > ixgbe_clean_tx_irq()
> > > >     -> if (!(eop_desc->wb.status & cpu_to_le32(IXGBE_TXD_STAT_DD)))
> > > >             break;
> > > > The above line always 'breaks' the sending process.
> > > >
> > > > I also managed to make the external ixgbe 6.15 work and it turned out
> > > > to be the same issue as before.
> > > >
> > > > I have no idea on how to analyze further in this driver. Could someone
> > > > point out a direction that I can take? Is it a known issue?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > >
> > > I was able to reproduce the described behaviour, xdpsock does break the IP
> > > communication. However, in my case this was not because of ixgbe not being able
> > > to send, but because of queue 0 RX packets being dropped, which is the indended
> > > outcome in xdpsock, even in Tx only mode.
> >
> > Thanks for your feedback. It would be great if you could elaborate
> > more on this. How did you spot that it's queue 0 that causes the
> > problem?
>
> If you do not specify -q parameter, xdpsock loads on the queue pair 0.
>
> > Why is xdpsock breaking IP communication intended?
>
> Because when a packet arrives on the AF_XDP-managed queue (0 in this case), the
> default xdpsock XDP program provided by libxdp returns XDP_REDIRECT even in
> tx-only mode, XDP_PASS for all other queues (1-39). XDP_REDIRECT results in a
> packet leaving the kernel network stack, it is now managed by the AF_XDP
> userspace program. I think it is possible to modify libxdp to return XDP_PASS
> when the socket is tx-only.
>
> >
> > When you try i40e, you will find the connection behaves normally. Ping
> > can work as usual. As I depicted before, with ixgbe driver, ping even
> > doesn't work at all.
>
> I think this is due to RSS configuration, ping packets on i40e go to another
> queue.

Thanks so much for your detailed explanation.

But, I still doubt it's not the reason why xdpsock breaks with ixgbe
driver loaded because I tried the following commands:
1. ping <ip>
2. timeout 3 ./xdpsock -i enp2s0f0np0 -t -q 11  -z -s 64
Whatever the queue value I adjusted, ping always fails with
"Destination Host Unreachable" warning.

Thanks,
Jason