Message-ID: <20240306100153.32d305f7@meshulam.tesarici.cz>
Date: Wed, 6 Mar 2024 10:01:53 +0100
From: Petr Tesařík <petr@...arici.cz>
To: "Linux regression tracking (Thorsten Leemhuis)"
 <regressions@...mhuis.info>
Cc: Linux regressions mailing list <regressions@...ts.linux.dev>, Eric
 Dumazet <edumazet@...gle.com>, "David S. Miller" <davem@...emloft.net>,
 Paolo Abeni <pabeni@...hat.com>, Jakub Kicinski <kuba@...nel.org>, Jisheng
 Zhang <jszhang@...nel.org>, Alexandre Torgue
 <alexandre.torgue@...s.st.com>, Jose Abreu <joabreu@...opsys.com>, Maxime
 Coquelin <mcoquelin.stm32@...il.com>, Chen-Yu Tsai <wens@...e.org>, Jernej
 Skrabec <jernej.skrabec@...il.com>, Samuel Holland <samuel@...lland.org>,
 "open list:STMMAC ETHERNET DRIVER" <netdev@...r.kernel.org>, "moderated
 list:ARM/STM32 ARCHITECTURE" <linux-stm32@...md-mailman.stormreply.com>,
 "moderated list:ARM/STM32 ARCHITECTURE"
 <linux-arm-kernel@...ts.infradead.org>, open list
 <linux-kernel@...r.kernel.org>, "open list:ARM/Allwinner sunXi SoC support"
 <linux-sunxi@...ts.linux.dev>, Marc Haber <mh+netdev@...schlus.de>, Andrew
 Lunn <andrew@...n.ch>, Florian Fainelli <f.fainelli@...il.com>,
 stable@...r.kernel.org, alexis.lothore@...tlin.com, Guenter Roeck
 <linux@...ck-us.net>
Subject: Re: [PATCH net v3] net: stmmac: protect updates of 64-bit
 statistics counters

On Wed, 6 Mar 2024 09:23:53 +0100
"Linux regression tracking (Thorsten Leemhuis)" <regressions@...mhuis.info> wrote:

> On 28.02.24 12:03, Petr Tesařík wrote:
> > On Wed, 28 Feb 2024 07:19:56 +0100
> > "Linux regression tracking (Thorsten Leemhuis)" <regressions@...mhuis.info> wrote:
> >   
> >> Net maintainers, chiming in here, as it seems handling this regression
> >> stalled.  
> > Indeed, I was too busy with sandbox mode...  
> 
> Hmm, no reply in the past week to Petr's request for help from someone
> with more knowledge about the field. :-/
> 
> So I guess this means that this won't be fixed for 6.8? Unfortunate, but
> well, that's how it it sometimes.

For the record, I _can_ reproduce lockdep splats on my device, but they
don't make any sense to me. They seem to confirm Jisheng Zhang's
conclusion that lockdep conflates two locks which should have different
lock-classes.

So far I have noticed only one issue: the per-cpu syncp's are not
initialized. I'll recompile and see if that's what confuses lockdep.

Petr T

> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
> 
> #regzbot poke
> 
> >> On 13.02.24 16:52, Eric Dumazet wrote:  
> >>> On Tue, Feb 13, 2024 at 4:26 PM Guenter Roeck <linux@...ck-us.net> wrote:    
> >>>> On Tue, Feb 13, 2024 at 03:51:35PM +0100, Eric Dumazet wrote:    
> >>>>> On Tue, Feb 13, 2024 at 3:29 PM Jisheng Zhang <jszhang@...nel.org> wrote:    
> >>>>>> On Sun, Feb 11, 2024 at 08:30:21PM -0800, Guenter Roeck wrote:    
> >>>>>>> On Sat, Feb 03, 2024 at 08:09:27PM +0100, Petr Tesarik wrote:    
> >>>>>>>> As explained by a comment in <linux/u64_stats_sync.h>, write side of struct
> >>>>>>>> u64_stats_sync must ensure mutual exclusion, or one seqcount update could
> >>>>>>>> be lost on 32-bit platforms, thus blocking readers forever. Such lockups
> >>>>>>>> have been observed in real world after stmmac_xmit() on one CPU raced with
> >>>>>>>> stmmac_napi_poll_tx() on another CPU.
> >>>>>>>>
> >>>>>>>> To fix the issue without introducing a new lock, split the statics into
> >>>>>>>> three parts:
> >>>>>>>>
> >>>>>>>> 1. fields updated only under the tx queue lock,
> >>>>>>>> 2. fields updated only during NAPI poll,
> >>>>>>>> 3. fields updated only from interrupt context,
> >>>>>>>>
> >>>>>>>> Updates to fields in the first two groups are already serialized through
> >>>>>>>> other locks. It is sufficient to split the existing struct u64_stats_sync
> >>>>>>>> so that each group has its own.
> >>>>>>>>
> >>>>>>>> Note that tx_set_ic_bit is updated from both contexts. Split this counter
> >>>>>>>> so that each context gets its own, and calculate their sum to get the total
> >>>>>>>> value in stmmac_get_ethtool_stats().
> >>>>>>>>
> >>>>>>>> For the third group, multiple interrupts may be processed by different CPUs
> >>>>>>>> at the same time, but interrupts on the same CPU will not nest. Move fields
> >>>>>>>> from this group to a newly created per-cpu struct stmmac_pcpu_stats.
> >>>>>>>>
> >>>>>>>> Fixes: 133466c3bbe1 ("net: stmmac: use per-queue 64 bit statistics where necessary")
> >>>>>>>> Link: https://lore.kernel.org/netdev/Za173PhviYg-1qIn@torres.zugschlus.de/t/
> >>>>>>>> Cc: stable@...r.kernel.org
> >>>>>>>> Signed-off-by: Petr Tesarik <petr@...arici.cz>    
> >>>>>>>
> >>>>>>> This patch results in a lockdep splat. Backtrace and bisect results attached.
> >>>>>>>
> >>>>>>> ---
> >>>>>>> [   33.736728] ================================
> >>>>>>> [   33.736805] WARNING: inconsistent lock state
> >>>>>>> [   33.736953] 6.8.0-rc4 #1 Tainted: G                 N
> >>>>>>> [   33.737080] --------------------------------
> >>>>>>> [   33.737155] inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
> >>>>>>> [   33.737309] kworker/0:2/39 [HC1[1]:SC0[2]:HE0:SE0] takes:
> >>>>>>> [   33.737459] ef792074 (&syncp->seq#2){?...}-{0:0}, at: sun8i_dwmac_dma_interrupt+0x9c/0x28c
> >>>>>>> [   33.738206] {HARDIRQ-ON-W} state was registered at:
> >>>>>>> [   33.738318]   lock_acquire+0x11c/0x368
> >>>>>>> [   33.738431]   __u64_stats_update_begin+0x104/0x1ac
> >>>>>>> [   33.738525]   stmmac_xmit+0x4d0/0xc58    
> >>>>>>
> >>>>>> interesting lockdep splat...
> >>>>>> stmmac_xmit() operates on txq_stats->q_syncp, while the
> >>>>>> sun8i_dwmac_dma_interrupt() operates on pcpu's priv->xstats.pcpu_stats
> >>>>>> they are different syncp. so how does lockdep splat happen.    
> >>>>>
> >>>>> Right, I do not see anything obvious yet.    
> >>>>
> >>>> Wild guess: I think it maybe saying that due to
> >>>>
> >>>>         inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage.
> >>>>
> >>>> the critical code may somehow be interrupted and, while handling the
> >>>> interrupt, try to acquire the same lock again.    
> >>>
> >>> This should not happen, the 'syncp' are different. They have different
> >>> lockdep classes.
> >>>
> >>> One is exclusively used from hard irq context.
> >>>
> >>> The second one only used from BH context.    
> >>
> >> Alexis Lothoré hit this now as well, see yesterday report in this
> >> thread; apart from that nothing seem to have happened for two weeks now.
> >> The change recently made it to some stable/longterm kernels, too. Makes
> >> me wonder:
> >>
> >> What's the plan forward here? Is this considered to be a false positive?  
> > 
> > Although my system has run stable for a couple of months, I am hesitant
> > to call it a false positive.
> >   
> >> Or a real problem?  
> > 
> > That's what I think. But I would have to learn a lot about the network
> > stack to understand what exactly happens here.
> > 
> > It may go faster if somebody else on the Cc can give me a hint where to
> > start looking based on the lockdep warning.
> > 
> > Petr T
> > 
> >   

Powered by blists - more mailing lists