netdev - Re: stmmac on Banana PI CPU stalls since Linux 6.6

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+h8R2okfaYn-=toQPCkQUEZ6oLuwfjZ0ZZ-zRiN9A2fBFmzHQ@mail.gmail.com>
Date: Mon, 19 Feb 2024 11:20:35 -0800
From: Christian Stewart <christian@...rture.us>
To: Marc Haber <mh+netdev@...schlus.de>
Cc: Petr Tesařík <petr@...arici.cz>, 
	Florian Fainelli <f.fainelli@...il.com>, Andrew Lunn <andrew@...n.ch>, alexandre.torgue@...s.st.com, 
	Jose Abreu <joabreu@...opsys.com>, Chen-Yu Tsai <wens@...e.org>, 
	Jernej Skrabec <jernej.skrabec@...il.com>, Samuel Holland <samuel@...lland.org>, 
	Jisheng Zhang <jszhang@...nel.org>, netdev@...r.kernel.org
Subject: Re: stmmac on Banana PI CPU stalls since Linux 6.6

Hi all,

On Mon, Feb 12, 2024 at 4:15 AM Marc Haber <mh+netdev@...schlus.de> wrote:
>
> On Tue, Feb 06, 2024 at 09:23:51AM +0100, Petr Tesařík wrote:
> > On Mon, 5 Feb 2024 13:50:35 -0800
> > Florian Fainelli <f.fainelli@...il.com> wrote:
> >
> > > On 2/5/24 12:12, Marc Haber wrote:
> > > > On Fri, Jan 26, 2024 at 12:10:28PM +0100, Petr Tesařík wrote:
> > > >> Then you may want to start by verifying that it is indeed the same
> > > >> issue. Try the linked patch.
> > > >
> > > > The linked patch seemed to help for 6.7.2, the test machine ran for five
> > > > days without problems. After going to unpatched 6.7.2, the issue was
> > > > back in six hours.
> > >
> > > Do you mind responding to Petr's patch with a Tested-by? Thanks!
> >
> > I believe Marc tested my first attempt at a solution (the one with
> > spinlocks), not the latest incarnation. FWIW I have tested a similar
> > scenario, with similar results.
>
> Where is the latest patch? I can give it a try.
>
> Sorry for not responding any earlier, February 10 is an important tax
> due date in Germany.
>
> Greetings
> Marc

We are seeing the same kernel panic on shutdown with 6.7.4 on a
BananaPi M2 Ultra:

[**    ] (3 of 3) A stop job is running for Network Manager (33s / 52s)
[  259.463772] rcu: INFO: rcu_sched self-detected stall on CPU
[  259.469388] rcu:     0-....: (2099 ticks this GP)
idle=0fdc/1/0x40000002 softirq=12003/12003 fqs=1034
[  259.478360] rcu:     (t=2100 jiffies g=16277 q=36 ncpus=4)
[  259.483595] CPU: 0 PID: 4462 Comm: ip Tainted: G         C         6.7.4 #1
[  259.490562] Hardware name: Allwinner sun8i Family
[  259.495268] PC is at stmmac_get_stats64+0x30/0x198
[  259.500081] LR is at dev_get_stats+0x3c/0x160
[  259.504445] pc : [<c06b9924>]    lr : [<c07bf7a8>]    psr: 200f0013
[  259.510712] sp : f1e6d9b8  ip : c3ca478c  fp : c23e0000
[  259.515941] r10: 00000000  r9 : c3ca4598  r8 : 00000000
[  259.521168] r7 : 00000001  r6 : 00000000  r5 : c23e3000  r4 : 00000001
[  259.527697] r3 : 00005c1b  r2 : c23e2e08  r1 : c3ca46c4  r0 : c23e0000
[  259.534226] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  259.541363] Control: 10c5387d  Table: 429cc06a  DAC: 00000051
[  259.547117]  stmmac_get_stats64 from dev_get_stats+0x3c/0x160
[  259.552882]  dev_get_stats from rtnl_fill_stats+0x30/0x118
[  259.552899]  rtnl_fill_stats from rtnl_fill_ifinfo+0x720/0x135c
[  259.564306]  rtnl_fill_ifinfo from rtnl_dump_ifinfo+0x330/0x6a8
[  259.570240]  rtnl_dump_ifinfo from netlink_dump+0x16c/0x350
[  259.575830]  netlink_dump from __netlink_dump_start+0x1bc/0x280
[  259.581766]  __netlink_dump_start from rtnetlink_rcv_msg+0xf4/0x2f0
[  259.588047]  rtnetlink_rcv_msg from netlink_rcv_skb+0xb8/0x118
[  259.593893]  netlink_rcv_skb from netlink_unicast+0x1fc/0x2d8
[  259.599655]  netlink_unicast from netlink_sendmsg+0x1c8/0x440
[  259.605416]  netlink_sendmsg from sock_write_iter+0xa0/0x10c
[  259.611094]  sock_write_iter from vfs_write+0x338/0x398
[  259.616334]  vfs_write from ksys_write+0xbc/0xf0
[  259.620961]  ksys_write from ret_fast_syscall+0x0/0x54
[  259.626110] Exception stack(0xf1e6dfa8 to 0xf1e6dff0)
[  259.631169] dfa0:                   00000003 be997dd8 00000003
be997dd8 00000014 00000001
[  259.639351] dfc0: 00000003 be997dd8 00000014 00000004 00519548
be997e08 b6fd0ce0 0051783c

https://github.com/skiffos/SkiffOS/issues/307

I'm writing to ask if anyone has found a fix for this yet?

Thanks!
Christian Stewart