| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20251202135832.156559-1-pmladek@suse.com>
Date: Tue, 2 Dec 2025 14:58:32 +0100
From: Petr Mladek <pmladek@...e.com>
To: John Ogness <john.ogness@...utronix.de>
Cc: Sergey Senozhatsky <senozhatsky@...omium.org>,
Steven Rostedt <rostedt@...dmis.org>,
Breno Leitao <leitao@...ian.org>,
linux@...linux.org.uk,
paulmck@...nel.org,
usamaarif642@...il.com,
leo.yan@....com,
linux-arm-kernel@...ts.infradead.org,
linux-kernel@...r.kernel.org,
kernel-team@...a.com,
rmikey@...a.com,
Petr Mladek <pmladek@...e.com>
Subject: [PATCH] printk/nbcon: Restore IRQ in atomic flush after each emitted record
The commit d5d399efff6577 ("printk/nbcon: Release nbcon consoles ownership
in atomic flush after each emitted record") prevented stall of a CPU
which lost nbcon console ownership because another CPU entered
an emergency flush.
But there is still the problem that the CPU doing the emergency flush
might cause a stall on its own.
Let's go even further and restore IRQ in the atomic flush after
each emitted record.
It is not a complete solution. The interrupts and/or scheduling might
still be blocked when the emergency atomic flush was called with
IRQs and/or scheduling disabled. But it should remove the following
lockup:
mlx5_core 0000:03:00.0: Shutdown was called
kvm: exiting hardware virtualization
arm-smmu-v3 arm-smmu-v3.10.auto: CMD_SYNC timeout at 0x00000103 [hwprod 0x00000104, hwcons 0x00000102]
smp: csd: Detected non-responsive CSD lock (#1) on CPU#4, waiting 5000000032 ns for CPU#00 do_nothing (kernel/smp.c:1057)
smp: csd: CSD lock (#1) unresponsive.
[...]
Call trace:
pl011_console_write_atomic (./arch/arm64/include/asm/vdso/processor.h:12 drivers/tty/serial/amba-pl011.c:2540) (P)
nbcon_emit_next_record (kernel/printk/nbcon.c:1049)
__nbcon_atomic_flush_pending_con (kernel/printk/nbcon.c:1517)
__nbcon_atomic_flush_pending.llvm.15488114865160659019 (./arch/arm64/include/asm/alternative-macros.h:254 ./arch/arm64/include/asm/cpufeature.h:808 ./arch/arm64/include/asm/irqflags.h:192 kernel/printk/nbcon.c:1562 kernel/printk/nbcon.c:1612)
nbcon_atomic_flush_pending (kernel/printk/nbcon.c:1629)
printk_kthreads_shutdown (kernel/printk/printk.c:?)
syscore_shutdown (drivers/base/syscore.c:120)
kernel_kexec (kernel/kexec_core.c:1045)
__arm64_sys_reboot (kernel/reboot.c:794 kernel/reboot.c:722 kernel/reboot.c:722)
invoke_syscall (arch/arm64/kernel/syscall.c:50)
el0_svc_common.llvm.14158405452757855239 (arch/arm64/kernel/syscall.c:?)
do_el0_svc (arch/arm64/kernel/syscall.c:152)
el0_svc (./arch/arm64/include/asm/alternative-macros.h:254 ./arch/arm64/include/asm/cpufeature.h:808 ./arch/arm64/include/asm/irqflags.h:73 arch/arm64/kernel/entry-common.c:169 arch/arm64/kernel/entry-common.c:182 arch/arm64/kernel/entry-common.c:749)
el0t_64_sync_handler (arch/arm64/kernel/entry-common.c:820)
el0t_64_sync (arch/arm64/kernel/entry.S:600)
In this case, nbcon_atomic_flush_pending() is called from
printk_kthreads_shutdown() with IRQs and scheduling enabled.
Note that __nbcon_atomic_flush_pending_con() is directly called also from
nbcon_device_release() where the disabled IRQs might break PREEMPT_RT
guarantees. But the atomic flush is called only in emergency or panic
situations where the latencies are irrelevant anyway.
An ultimate solution would be a touching of watchdogs. But it would hide
all problems. Let's do it later when anyone reports a stall which does
not have a better solution.
Closes: https://lore.kernel.org/r/sqwajvt7utnt463tzxgwu2yctyn5m6bjwrslsnupfexeml6hkd@v6sqmpbu3vvu
Tested-by: Breno Leitao <leitao@...ian.org>
Signed-off-by: Petr Mladek <pmladek@...e.com>
---
Hi,
I sent this patch for a proper review. It has been sent as RFC
at https://lore.kernel.org/all/aSnI8UQRNICSKxAb@pathway.suse.cz/
Changes against RFC:
+ Added note about __nbcon_atomic_flush_pending_con() called from
nbcon_device_release() and PREEMPT_RT into the commit message [John].
+ Added Tested-by [Breno]
kernel/printk/nbcon.c | 29 ++++++++++++++++-------------
1 file changed, 16 insertions(+), 13 deletions(-)
diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index 3fa403f9831f..6b8becb6ecd9 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1549,6 +1549,7 @@ static int __nbcon_atomic_flush_pending_con(struct console *con, u64 stop_seq)
{
struct nbcon_write_context wctxt = { };
struct nbcon_context *ctxt = &ACCESS_PRIVATE(&wctxt, ctxt);
+ unsigned long flags;
int err = 0;
ctxt->console = con;
@@ -1557,18 +1558,31 @@ static int __nbcon_atomic_flush_pending_con(struct console *con, u64 stop_seq)
ctxt->allow_unsafe_takeover = nbcon_allow_unsafe_takeover();
while (nbcon_seq_read(con) < stop_seq) {
- if (!nbcon_context_try_acquire(ctxt, false))
+ /*
+ * Atomic flushing does not use console driver synchronization
+ * (i.e. it does not hold the port lock for uart consoles).
+ * Therefore IRQs must be disabled to avoid being interrupted
+ * and then calling into a driver that will deadlock trying
+ * to acquire console ownership.
+ */
+ local_irq_save(flags);
+ if (!nbcon_context_try_acquire(ctxt, false)) {
+ local_irq_restore(flags);
return -EPERM;
+ }
/*
* nbcon_emit_next_record() returns false when the console was
* handed over or taken over. In both cases the context is no
* longer valid.
*/
- if (!nbcon_emit_next_record(&wctxt, true))
+ if (!nbcon_emit_next_record(&wctxt, true)) {
+ local_irq_restore(flags);
return -EAGAIN;
+ }
nbcon_context_release(ctxt);
+ local_irq_restore(flags);
if (!ctxt->backlog) {
/* Are there reserved but not yet finalized records? */
@@ -1595,22 +1609,11 @@ static int __nbcon_atomic_flush_pending_con(struct console *con, u64 stop_seq)
static void nbcon_atomic_flush_pending_con(struct console *con, u64 stop_seq)
{
struct console_flush_type ft;
- unsigned long flags;
int err;
again:
- /*
- * Atomic flushing does not use console driver synchronization (i.e.
- * it does not hold the port lock for uart consoles). Therefore IRQs
- * must be disabled to avoid being interrupted and then calling into
- * a driver that will deadlock trying to acquire console ownership.
- */
- local_irq_save(flags);
-
err = __nbcon_atomic_flush_pending_con(con, stop_seq);
- local_irq_restore(flags);
-
/*
* If there was a new owner (-EPERM, -EAGAIN), that context is
* responsible for completing.
--
2.52.0
Powered by blists - more mailing lists