Message-ID: <Z3Kuu44L0ZcnavQF@LQ3V64L9R2>
Date: Mon, 30 Dec 2024 09:31:23 -0500
From: Joe Damato <jdamato@...tly.com>
To: Alex Lazar <alazar@...dia.com>
Cc: "aleksander.lobakin@...el.com" <aleksander.lobakin@...el.com>,
	"almasrymina@...gle.com" <almasrymina@...gle.com>,
	"amritha.nambiar@...el.com" <amritha.nambiar@...el.com>,
	"bigeasy@...utronix.de" <bigeasy@...utronix.de>,
	"bjorn@...osinc.com" <bjorn@...osinc.com>,
	"corbet@....net" <corbet@....net>, Dan Jurgens <danielj@...dia.com>,
	"davem@...emloft.net" <davem@...emloft.net>,
	"donald.hunter@...il.com" <donald.hunter@...il.com>,
	"dsahern@...nel.org" <dsahern@...nel.org>,
	"edumazet@...gle.com" <edumazet@...gle.com>,
	"hawk@...nel.org" <hawk@...nel.org>,
	"jiri@...nulli.us" <jiri@...nulli.us>,
	"johannes.berg@...el.com" <johannes.berg@...el.com>,
	"kuba@...nel.org" <kuba@...nel.org>,
	"leitao@...ian.org" <leitao@...ian.org>,
	"leon@...nel.org" <leon@...nel.org>,
	"linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
	"lorenzo@...nel.org" <lorenzo@...nel.org>,
	"michael.chan@...adcom.com" <michael.chan@...adcom.com>,
	"mkarsten@...terloo.ca" <mkarsten@...terloo.ca>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"pabeni@...hat.com" <pabeni@...hat.com>,
	Saeed Mahameed <saeedm@...dia.com>,
	"sdf@...ichev.me" <sdf@...ichev.me>,
	"skhawaja@...gle.com" <skhawaja@...gle.com>,
	"sridhar.samudrala@...el.com" <sridhar.samudrala@...el.com>,
	Tariq Toukan <tariqt@...dia.com>,
	"willemdebruijn.kernel@...il.com" <willemdebruijn.kernel@...il.com>,
	"xuanzhuo@...ux.alibaba.com" <xuanzhuo@...ux.alibaba.com>,
	Gal Pressman <gal@...dia.com>, Nimrod Oren <noren@...dia.com>,
	Dror Tennenbaum <drort@...dia.com>,
	Dragos Tatulea <dtatulea@...dia.com>
Subject: Re: [net-next v6 0/9] Add support for per-NAPI config via netlink

On Mon, Dec 23, 2024 at 08:17:08AM +0000, Alex Lazar wrote:
> 
> 
> On 20/12/2024 19:40, Joe Damato wrote:
> > On Wed, Dec 18, 2024 at 09:08:58AM -0800, Joe Damato wrote:
> >> On Wed, Dec 18, 2024 at 11:22:33AM +0000, Alex Lazar wrote:
> >>> Hi Joe and all,
> >>>
> >>> I am part of the NVIDIA Eth drivers team, and we are experiencing a problem,
> >>> sibesced to this change: commit 86e25f40aa1e ("net: napi: Add napi_config")
> >>>
> >>> The issue occurs when sending packets from one machine to another.
> >>> On the receiver side, we have XSK (XDPsock) that receives the packet and sends it
> >>> back to the sender.
> >>> At some point, one packet (packet A) gets "stuck," and if we send a new packet
> >>> (packet B), it "pushes" the previous one. Packet A is then processed by the NAPI
> >>> poll, and packet B gets stuck, and so on.
> >>>
> >>> Your change involves moving napi_hash_del() and napi_hash_add() from
> >>> netif_napi_del() and netif_napi_add_weight() to napi_enable() and napi_disable().
> >>> If I move them back to netif_napi_del() and netif_napi_add_weight(),
> >>> the issue is resolved (I moved the entire if/else block, not just the napi_hash_del/add).
> >>>
> >>> This issue occurs with both the new and old APIs (netif_napi_add/_config).
> >>> Moving the napi_hash_add() and napi_hash_del() functions resolves it for both.
> >>> I am debugging this, no breakthrough so far.
> >>>
> >>> I would appreciate if you could look into this.
> >>> We can provide more details per request.
> >>
> >> I appreciate your report, but there is not a lot in your message to
> >> help debug the issue.
> >>
> >> Can you please:
> >>
> >> 1.) Verify that the kernel tree you are testing on has commit
> >> cecc1555a8c2 ("net: Make napi_hash_lock irq safe") included ? If it
> >> does not, can you pull in that commit and re-run your test and
> >> report back if that fixes your problem?
> 
> I verified that the kernel tree includes commit cecc1555a8c2 ("net: Make 
> napi_hash_lock irq safe"), but the issue still occurs.

OK, thanks for verifying that.
 
> >>
> >> 2.) If (1) does not fix your problem, can you please reply with at
> >> least the following information:
> >>    - Specify what device this is happening on (in case I have access
> >>      to one)
> 
> We are using two ConnectX-5 cards connected back-to-back.
> 
> >>    - Which driver is affected
> 
> The affected driver is the MLX5 driver.
> 
> >>    - Which upstream kernel SHA you are building your test kernel from
> 
> The upstream kernel SHA we are building is 9163b05eca1d ("Merge branch 
> 'add-support-for-so_priority-cmsg'").

I have access to ConnectX-5 Ex EN MCX516A-CDAT NICs and can build a
kernel based on the upstream SHA you mentioned.

> >>    - The reproducer program(s) with clear instructions on how exactly
> >>      to run it/them in order to reproduce the issue
> 
> Test setup/configuration:
> On one side, we use a Python script with the scapy.all library to create 
> UDP packets of size 1024, using port 19017 and the MAC/IP of the other side.
> On the other side, we define an n-tuple filter (ethtool --config-ntuple 
> eth2 flow-type udp4 dst-port 19017 action 4) and run xdpsock (xdpsock -i 
> eth2 -N -q 4 --l2fwd -z -B).
> In the test, we send a single packet each time, which is received and 
> sent back to the sender.
> As part of the validation, we check the statistics on the other side and 
> notice a discrepancy between what xdpsock shows and what we see in the 
> driver (ethtool -S eth2 | grep "tx_xsk_xmit").

1. Can you please be much more specific when you say "notice a
   discrepancy" and substitute that for specific data? What,
   exactly, is a single run of the experiment that results in a lost
   or delayed packet? Do you have tcpdump output? What is the
   expected output and what is the actual output?

2. Can you provide the full source code? Both the reproducer and the
   script to check the values?

   You can feel free to create a github gist and link to it, then I
   can git clone it and run it.

   In the gist you can create a README.md that shows exactly what
   commands I need to run to reproduce, the expected output from the
   script, and the actual output.

   Since I have access to this hardware, I should be able to
   reproduce.

   Ideally, I'll git clone your gist, run something like "make test"
   (or similar) and reproduce the issue and get either "Test
   succeeded" or Test failed; expected %d packets but received %d".

3. If this issue is with napi_disable/napi_enable, can you think of
   a simpler reproducer? For example, wouldn't changing the queue
   count cause the same issue without needing to involve xdp at all?

   What about a simpler experiment with UDPing [1] and an ntuple
   filter? Does it reproduce with plain old NAPI processing or is it
   XDP specific?

> > 
> > I didn't hear back on the above, but wanted to let you know that
> > I'll be out of the office soon, so my responses/bandwidth for
> > helping to debug this will be limited over the next week or two.
> 
> Hi Joe,
> 
> Thanks for the quick response.
> Comments inline, If you need more details or further clarification, 
> please let me know.

As mentioned above and in my previous emails: please provide lot
more detail and make it as easy as possible for me to reproduce this
issue with the simplest reproducer possible and a much more detailed
explanation.

Please note: I will be out of the office until Jan 9 so my responses
will be limited until then.

Powered by blists - more mailing lists