netdev - Re: [PATCH net] sfc: use budget for TX completions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20230614102744.71c91f20@kernel.org>
Date: Wed, 14 Jun 2023 10:27:44 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: Martin Habets <habetsm.xilinx@...il.com>
Cc: Íñigo Huguet <ihuguet@...hat.com>,
 ecree.xilinx@...il.com, davem@...emloft.net, edumazet@...gle.com,
 pabeni@...hat.com, netdev@...r.kernel.org, linux-net-drivers@....com, Fei
 Liu <feliu@...hat.com>
Subject: Re: [PATCH net] sfc: use budget for TX completions

On Wed, 14 Jun 2023 09:03:05 +0100 Martin Habets wrote:
> On Mon, Jun 12, 2023 at 04:42:54PM +0200, Íñigo Huguet wrote:
> > When running workloads heavy unbalanced towards TX (high TX, low RX
> > traffic), sfc driver can retain the CPU during too long times. Although
> > in many cases this is not enough to be visible, it can affect
> > performance and system responsiveness.
> > 
> > A way to reproduce it is to use a debug kernel and run some parallel
> > netperf TX tests. In some systems, this will lead to this message being
> > logged:
> >   kernel:watchdog: BUG: soft lockup - CPU#12 stuck for 22s!
> > 
> > The reason is that sfc driver doesn't account any NAPI budget for the TX
> > completion events work. With high-TX/low-RX traffic, this makes that the
> > CPU is held for long time for NAPI poll.
> > 
> > Documentations says "drivers can process completions for any number of Tx
> > packets but should only process up to budget number of Rx packets".
> > However, many drivers do limit the amount of TX completions that they
> > process in a single NAPI poll.  
> 
> I think your work and what other drivers do shows that the documentation is
> no longer correct. I haven't checked when that was written, but maybe it
> was years ago when link speeds were lower.
> Clearly for drivers that support higher link speeds this is an issue, so we
> should update the documentation. Not sure what constitutes a high link speed,
> with current CPUs for me it's anything >= 50G.

The documentation is pretty recent. I haven't seen this lockup once 
in production or testing. Do multiple queues complete on the same CPU
for SFC or something weird like that?