linux-kernel - Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <41f07451-8d77-460d-aff8-fadce111a511@gmail.com>
Date: Tue, 12 Mar 2024 14:35:15 -0700
From: Florian Fainelli <f.fainelli@...il.com>
To: paulmck@...nel.org, Frederic Weisbecker <frederic@...nel.org>
Cc: Boqun Feng <boqun.feng@...il.com>, torvalds@...ux-foundation.org,
 linux-kernel@...r.kernel.org, kernel-team@...a.com, mingo@...nel.org,
 tglx@...utronix.de, rcu@...r.kernel.org, joel@...lfernandes.org,
 neeraj.upadhyay@....com, urezki@...il.com, qiang.zhang1211@...il.com,
 bigeasy@...utronix.de, anna-maria@...utronix.de, chenzhongjin@...wei.com,
 yangjihong1@...wei.com, rostedt@...dmis.org
Subject: Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for
 v6.9]

On 3/12/24 14:15, Paul E. McKenney wrote:
> On Tue, Mar 12, 2024 at 10:01:00PM +0100, Frederic Weisbecker wrote:
>> Le Tue, Mar 12, 2024 at 01:32:03PM -0700, Florian Fainelli a écrit :
>>> Hi Boqun,
>>>
>>> On 3/8/24 09:15, Boqun Feng wrote:
>>>> Hi Linus,
>>>>
>>>> Please pull this for the RCU changes of v6.9:
>>>>
>>>> The following changes since commit 41bccc98fb7931d63d03f326a746ac4d429c1dd3:
>>>>
>>>>     Linux 6.8-rc2 (2024-01-28 17:01:12 -0800)
>>>>
>>>> are available in the Git repository at:
>>>>
>>>>     git://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git tags/rcu.next.v6.9
>>>>
>>>> for you to fetch changes up to 3add00be5fe5810d7aa5ec3af8b6a245ef33144b:
>>>>
>>>>     Merge branches 'rcu-doc.2024.02.14a', 'rcu-nocb.2024.02.14a', 'rcu-exp.2024.02.14a', 'rcu-tasks.2024.02.26a' and 'rcu-misc.2024.02.14a' into rcu.2024.02.26a (2024-02-26 17:37:25 -0800)
>>>>
>>>>
>>>> Two merge conflicts were detected by linux-next:
>>>>
>>>> * https://lore.kernel.org/lkml/20240226135745.12ac854d@canb.auug.org.au/
>>>> * https://lore.kernel.org/lkml/20240227125522.2bdbe6be@canb.auug.org.au/
>>>>
>>>> These conflict resolutions from linux-next look good to me, plus I made
>>>> my own resolutions at branch merge/rcu.2024.02.27a for your reference.
>>>>
>>>>
>>>> Some highlights of the changes:
>>>>
>>>> * Eliminates deadlocks involving do_exit() and RCU tasks, by Paul:
>>>>     Instead of SRCU read side critical sections, now a percpu list is used
>>>>     in do_exit() for scaning yet-to-exit tasks.
>>>>
>>>> * Fixes a deadlock due to the dependency between workqueue and RCU
>>>>     expedited grace period, reported by Anna-Maria Behnsen and Thomas
>>>>     Gleixner and fixed by Frederic: Now RCU expedited always uses its own
>>>>     kthread worker instead of a workqueue.
>>>
>>> At least one device in my test farm (ARM 32-bit) has consistently shown a
>>> very long boot, and some others are intermittently affected. This
>>> consistently looks like this on most of my devices:
>>>
>>> [    2.450351] bcmgenet f0480000.ethernet: GENET 5.0 EPHY: 0x0000
>>> [    2.547562] ata1: SATA link down (SStatus 0 SControl 300)
>>> [  162.107264] unimac-mdio unimac-mdio.0: Broadcom UniMAC MDIO bus
>>>
>>> this gets flagged by my boot script as a boot failure since we exceeded the
>>> 30 seconds timeout given to boot a kernel to a prompt.
>>>
>>> It has been somewhat difficult to get a reliable bisection going on, but
>>> what I am sure of is that e5a3878c947ceef7b6ab68fdc093f3848059842c~1 does
>>> not expose the problem for 10 consecutive boots, while I *might* see it at
>>> e5a3878c947ceef7b6ab68fdc093f3848059842c and beyond.
>>>
>>> Any clues what is going on here?
>>
>> I understand it doesn't reliably reproduce. But is there any chance you could
>> test that alone?
>>
>>       0bb11a372fc8 (rcu-tasks: Maintain real-time response in rcu_tasks_postscan())
>>
>> And if it reproduces there try a bisection?
>>
>> git bisect bad 0bb11a372fc8d7006b4d0f42a2882939747bdbff
>> git bisect good 41bccc98fb7931d63d03f326a746ac4d429c1dd3
> 
> Good point!
> 
> Is this kernel built with CONFIG_PREEMPT_RT=y and CONFIG_TASKS_RCU=y?
> Either way, is there also a large number of tasks exiting at this point
> during boot?

Neither options are enabled, and this is still during boot, so there is 
not any user-space tasks being running yet, I cannot think of too many 
reasons why this would fail, and there is no driver on the probe 
deferral list that I can see.
-- 
Florian