[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-Id: <20220126230236.750229-1-stephen.s.brennan@oracle.com>
Date: Wed, 26 Jan 2022 15:02:32 -0800
From: Stephen Brennan <stephen.s.brennan@...cle.com>
To: Andrew Morton <akpm@...ux-foundation.org>,
Sergey Senozhatsky <senozhatsky@...omium.org>,
Petr Mladek <pmladek@...e.com>
Cc: Arnd Bergmann <arnd@...db.de>,
Steven Rostedt <rostedt@...dmis.org>,
Stephen Brennan <stephen.s.brennan@...cle.com>,
Sebastian Reichel <sre@...nel.org>,
John Ogness <john.ogness@...utronix.de>,
Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
Luis Chamberlain <mcgrof@...nel.org>,
linux-kernel@...r.kernel.org
Subject: [PATCH v2 0/4] printk: reduce deadlocks during panic
When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).
The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.
However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.
A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.
To test, I use the following script, kmsg_panic.sh:
#!/bin/bash
date
# 991 chars (based on log buffer size):
chars="$(printf 'a%.0s' {1..991})"
while :; do
echo $chars > /dev/kmsg
done &
echo c > /proc/sysrq-trigger &
date
exit
I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.
before: 776 hangs / 1484 trials - 52.3%
after : 0 hangs / 408 trials - 0.0%
v2:
Each patch has minor updates from code reviews. I've re-done testing and
updated the above statistics. Exact changes are in each patch.
Stephen Brennan (4):
panic: Add panic_in_progress helper
printk: disable optimistic spin during panic
printk: Avoid livelock with heavy printk during panic
printk: Drop console_sem during panic
include/linux/panic.h | 6 +++++
kernel/printk/printk.c | 51 +++++++++++++++++++++++++++++++++++++++++-
2 files changed, 56 insertions(+), 1 deletion(-)
--
2.30.2
Powered by blists - more mailing lists