Message-ID: <CA+G9fYvcb-BAreFW=QS=DnWnXX1gK1EAX09tBQzopwH1Jd1d=w@mail.gmail.com>
Date: Fri, 11 Jul 2025 12:10:57 +0530
From: Naresh Kamboju <naresh.kamboju@...aro.org>
To: rcu <rcu@...r.kernel.org>, open list <linux-kernel@...r.kernel.org>, 
	lkft-triage@...ts.linaro.org, Linux Regressions <regressions@...ts.linux.dev>
Cc: "Paul E. McKenney" <paulmck@...nel.org>, neeraj.upadhyay@...nel.org, 
	Joel Fernandes <joelagnelf@...dia.com>, Qi Xi <xiqi2@...wei.com>, 
	Xiongfeng Wang <wangxiongfeng2@...wei.com>, Dan Carpenter <dan.carpenter@...aro.org>, 
	Arnd Bergmann <arnd@...db.de>, Anders Roxell <anders.roxell@...aro.org>, 
	Ben Copeland <benjamin.copeland@...aro.org>
Subject: next-20250710: PREEMPT_RT: rcu_preempt self-detected stall on CPU rcu_preempt_deferred_qs_handler

Regressions on boot failure observed in builds with CONFIG_PREEMPT_RT=y
and CONFIG_LAZY_PREEMPT=y enabled on multiple platforms running the
Linux next-20250710.

Kernel stalls and RCU preempt self-detected stalls reported.
Boot log shows CPU stalls and failure as below.

Test environments:
- Ampere Altra
- rk3399-rock-pi-4b
- x86_64

Regression Analysis:
- New regression? Yes
- Reproducibility? Yes

Boot regression: next-20250710 rcu_preempt self-detected stall on CPU
rcu_preempt_deferred_qs_handler

Reported-by: Linux Kernel Functional Testing <lkft@...aro.org>

Anders bisected this down to,
  first bad commit:
    [3284e4adca9b5b5a9f58dd071b848629e3bbecf8]
    rcu: Fix rcu_read_unlock() deadloop due to IRQ work

## Boot log
[ 12.001815] check access for rdinit=/init failed: -2, ignoring
[   22.252627] platform sdio-pwrseq: deferred probe pending:
pwrseq_simple: reset control not ready
[   22.252653] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to fe310000.mmc
[   22.252663] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff650000.video-codec
[   22.252672] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff660000.video-codec
[   22.252681] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff680000.rga
[   22.252696] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff880000.i2s
[   22.252704] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff8a0000.i2s
[   22.252712] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff8f0000.vop
[   22.252721] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff900000.vop
[   22.252735] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff940000.hdmi
[   22.252743] rockchip-pm-domain
ff310000.power-management:power-controller: sync_state() pending due
to ff9a0000.gpu
[   22.257625] dwmmc_rockchip fe310000.mmc: IDMAC supports 32-bit address mode.
[   22.257666] dwmmc_rockchip fe310000.mmc: Using internal DMA controller.
[   22.257678] dwmmc_rockchip fe310000.mmc: Version ID is 270a
[   22.258153] dwmmc_rockchip fe310000.mmc: DW MMC controller at irq
49,32 bit host data width,256 deep fifo
[   23.142949] rcu: INFO: rcu_preempt self-detected stall on CPU
[   23.142962] rcu: t0-....: (5196 ticks this GP)
idle=195c/1/0x4000000000000000 softirq=0/0 fqs=2620 rcuc=5196
jiffies(starved)
[   23.142973] rcu: t(t=5250 jiffies g=-887 q=360 ncpus=6)
[   23.142984] CPU: 0 UID: 0 PID: 24 Comm: irq_work/0 Not tainted
6.16.0-rc5-next-20250710 #1 PREEMPT_RT
[   23.142991] Hardware name: Radxa ROCK Pi 4B (DT)
[   23.142995] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   23.143002] pc : rcu_preempt_deferred_qs_handler
(kernel/rcu/tree_plugin.h:647)
[   23.143020] lr : rcu_preempt_deferred_qs_handler
(arch/arm64/include/asm/irqflags.h:175
arch/arm64/include/asm/irqflags.h:195 kernel/rcu/tree_plugin.h:646)
[   23.143029] sp : ffff80008329bda0
[   23.143031] x29: ffff80008329bda0 x28: 0000000000000000 x27: 0000000000000000
[   23.143043] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00000061e320
[   23.143052] x23: ffff000000cd5f00 x22: ffff000000cd5f00 x21: 0000000000000020
[   23.143061] x20: 0000000000000000 x19: ffff0000f750f078 x18: 0000000000000000
[   23.143069] x17: 0000000000000000 x16: ffff800081cea140 x15: 0000000000000033
[   23.143078] x14: 0000000000000000 x13: e800521362c92235 x12: aa42a37f93df442b
[   23.143087] x11: ffff800083e2b730 x10: 0000000000000c40 x9 : ffff8000815d24dc
[   23.143096] x8 : ffff80008329bc88 x7 : 0000000000000000 x6 : ffff8000828ce000
[   23.143105] x5 : ffff000000cd5f00 x4 : ffff8000828ce3a0 x3 : 0000000000000000
[   23.143113] x2 : 0000000000000000 x1 : 0000000100000000 x0 : 0000000100000000
[   23.143122] Call trace:
[   23.143126] rcu_preempt_deferred_qs_handler
(kernel/rcu/tree_plugin.h:647) (P)
[   23.143137] irq_work_single (kernel/irq_work.c:222 (discriminator 1))
[   23.143149] irq_work_run_list.part.0 (kernel/irq_work.c:251
(discriminator 1))
[   23.143159] run_irq_workd (kernel/irq_work.c:306)
[   23.143168] smpboot_thread_fn (kernel/smpboot.c:160)
[   23.143179] kthread (kernel/kthread.c:463)
[   23.143189] ret_from_fork (arch/arm64/kernel/entry.S:859)
[   28.908093] VFS: Mounted root (nfs filesystem) on device 0:23.
[   28.908665] devtmpfs: mounted
[   28.923586] Freeing unused kernel memory: 4160K
[   28.923786] Run /sbin/init as init process
[   49.938947] rcu: INFO: rcu_preempt self-detected stall on CPU
[   49.938955] rcu: t4-....: (1 GPs behind)
idle=0aec/1/0x4000000000000000 softirq=0/0 fqs=2617 rcuc=11947
jiffies(starved)
[   49.938963] rcu: t(t=5251 jiffies g=-883 q=3276 ncpus=6)
[   49.938971] CPU: 4 UID: 0 PID: 51 Comm: irq_work/4 Not tainted
6.16.0-rc5-next-20250710 #1 PREEMPT_RT
[   49.938976] Hardware name: Radxa ROCK Pi 4B (DT)
[   49.938978] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   49.938982] pc : rcu_preempt_deferred_qs_handler
(kernel/rcu/tree_plugin.h:647)
[   49.938998] lr : rcu_preempt_deferred_qs_handler
(arch/arm64/include/asm/irqflags.h:175
arch/arm64/include/asm/irqflags.h:195 kernel/rcu/tree_plugin.h:646)
[   49.939003] sp : ffff80008339bda0
[   49.939004] x29: ffff80008339bda0 x28: 0000000000000000 x27: 0000000000000000
[   49.939012] x26: 0000000000000000 x25: 0000000000000000 x24: ffff00000061e5a0
[   49.939017] x23: ffff000000ed9300 x22: ffff000000ed9300 x21: 0000000000000020
[   49.939023] x20: 0000000000000000 x19: ffff0000f7597078 x18: ffff8000837fb388
[   49.939028] x17: ffff800082d35920 x16: 0000000000000000 x15: 000000007794abb1
[   49.939033] x14: 0000000000000001 x13: 6463682d69636878 x12: 0000000000000001
[   49.939038] x11: ffff0000f75961c0 x10: 0000000000000c40 x9 : ffff8000815d24dc
[   49.939043] x8 : ffff80008339bc88 x7 : 0000000000000000 x6 : ffff8000828ce000
[   49.939048] x5 : ffff000000ed9300 x4 : ffff8000828ce3a0 x3 : 0000000000000000
[   49.939053] x2 : 0000000000000000 x1 : 0000000100000000 x0 : 0000000100000000
[   49.939059] Call trace:
[   49.939061] rcu_preempt_deferred_qs_handler
(kernel/rcu/tree_plugin.h:647) (P)
[   49.939068] irq_work_single (kernel/irq_work.c:222 (discriminator 1))
[   49.939076] irq_work_run_list.part.0 (kernel/irq_work.c:251
(discriminator 1))
[   49.939082] run_irq_workd (kernel/irq_work.c:306)
[   49.939087] smpboot_thread_fn (kernel/smpboot.c:160)
[   49.939094] kthread (kernel/kthread.c:463)
[   49.939101] ret_from_fork (arch/arm64/kernel/entry.S:859)
[   65.639064] random: crng init done

## Source
* Kernel version: 6.16.0-rc5-next-20250710
* Git tree: https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git
* Git sha: b551c4e2a98a177a06148cf16505643cd2108386
* Git describe: next-20250710
* Project: https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20250710
* Architectures: arm64
* Toolchains: gcc-13
* Kconfigs: gcc-13-defconfig-preempt_rt

## Build
* Test log: https://qa-reports.linaro.org/api/testruns/29088924/log_file/
* Test LAVA log 1: https://lkft.validation.linaro.org/scheduler/job/8351708#L947
* Test details:
https://regressions.linaro.org/lkft/linux-next-master/next-20250710/boot/gcc-13-defconfig-preempt_rt/
* Test plan: https://tuxapi.tuxsuite.com/v1/groups/linaro/projects/lkft/tests/2zg2ZkIKDYPorIm1qfg2NEzlIkR
* Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2zg2YGdS8dEcQIkEuzXik2IqIss/
* Kernel config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2zg2YGdS8dEcQIkEuzXik2IqIss/config

--
Linaro LKFT
https://lkft.linaro.org

Powered by blists - more mailing lists