| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20251016170212.65e2ad95b80cdeeb6f7d7ce3@kernel.org>
Date: Thu, 16 Oct 2025 17:02:12 +0900
From: Masami Hiramatsu (Google) <mhiramat@...nel.org>
To: lirongqing <lirongqing@...du.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>, Lance Yang
<lance.yang@...ux.dev>, <linux-kernel@...r.kernel.org>,
<linux-doc@...r.kernel.org>, <linux-arm-kernel@...ts.infradead.org>,
<linux-aspeed@...ts.ozlabs.org>, <wireguard@...ts.zx2c4.com>,
<netdev@...r.kernel.org>, <linux-kselftest@...r.kernel.org>, Andrew Jeffery
<andrew@...econstruct.com.au>, Anshuman Khandual
<anshuman.khandual@....com>, Arnd Bergmann <arnd@...db.de>, David
Hildenbrand <david@...hat.com>, Florian Wesphal <fw@...len.de>, Jakub
Kacinski <kuba@...nel.org>, "Jason A . Donenfeld" <jason@...c4.com>, Joel
Granados <joel.granados@...nel.org>, Joel Stanley <joel@....id.au>,
Jonathan Corbet <corbet@....net>, Kees Cook <kees@...nel.org>, Liam Howlett
<liam.howlett@...cle.com>, Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
"Paul E . McKenney" <paulmck@...nel.org>, Pawan Gupta
<pawan.kumar.gupta@...ux.intel.com>, Petr Mladek <pmladek@...e.com>, Phil
Auld <pauld@...hat.com>, Randy Dunlap <rdunlap@...radead.org>, Russell King
<linux@...linux.org.uk>, Shuah Khan <shuah@...nel.org>, Simon Horman
<horms@...nel.org>, Stanislav Fomichev <sdf@...ichev.me>, Steven Rostedt
<rostedt@...dmis.org>
Subject: Re: [PATCH][v4] hung_task: Panic when there are more than N hung
tasks at the same time
On Wed, 15 Oct 2025 14:36:15 +0800
lirongqing <lirongqing@...du.com> wrote:
> From: Li RongQing <lirongqing@...du.com>
>
> Currently, when 'hung_task_panic' is enabled, the kernel panics
> immediately upon detecting the first hung task. However, some hung
> tasks are transient and allow system recovery, while persistent hangs
> should trigger a panic when accumulating beyond a threshold.
>
> Extend the 'hung_task_panic' sysctl to accept a threshold value
> specifying the number of hung tasks that must be detected before
> triggering a kernel panic. This provides finer control for environments
> where transient hangs may occur but persistent hangs should be fatal.
>
> The sysctl now accepts:
> - 0: don't panic (maintains original behavior)
> - 1: panic on first hung task (maintains original behavior)
> - N > 1: panic after N hung tasks are detected in a single scan
>
> This maintains backward compatibility while providing flexibility for
> different hang scenarios.
Looks good to me.
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@...nel.org>
Thank you,
>
> Signed-off-by: Li RongQing <lirongqing@...du.com>
> Cc: Andrew Jeffery <andrew@...econstruct.com.au>
> Cc: Anshuman Khandual <anshuman.khandual@....com>
> Cc: Arnd Bergmann <arnd@...db.de>
> Cc: David Hildenbrand <david@...hat.com>
> Cc: Florian Wesphal <fw@...len.de>
> Cc: Jakub Kacinski <kuba@...nel.org>
> Cc: Jason A. Donenfeld <jason@...c4.com>
> Cc: Joel Granados <joel.granados@...nel.org>
> Cc: Joel Stanley <joel@....id.au>
> Cc: Jonathan Corbet <corbet@....net>
> Cc: Kees Cook <kees@...nel.org>
> Cc: Lance Yang <lance.yang@...ux.dev>
> Cc: Liam Howlett <liam.howlett@...cle.com>
> Cc: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
> Cc: "Masami Hiramatsu (Google)" <mhiramat@...nel.org>
> Cc: "Paul E . McKenney" <paulmck@...nel.org>
> Cc: Pawan Gupta <pawan.kumar.gupta@...ux.intel.com>
> Cc: Petr Mladek <pmladek@...e.com>
> Cc: Phil Auld <pauld@...hat.com>
> Cc: Randy Dunlap <rdunlap@...radead.org>
> Cc: Russell King <linux@...linux.org.uk>
> Cc: Shuah Khan <shuah@...nel.org>
> Cc: Simon Horman <horms@...nel.org>
> Cc: Stanislav Fomichev <sdf@...ichev.me>
> Cc: Steven Rostedt <rostedt@...dmis.org>
> ---
> diff with v3: comments modification, suggested by Lance, Masami, Randy and Petr
> diff with v2: do not add a new sysctl, extend hung_task_panic, suggested by Kees Cook
>
> Documentation/admin-guide/kernel-parameters.txt | 20 +++++++++++++-------
> Documentation/admin-guide/sysctl/kernel.rst | 9 +++++----
> arch/arm/configs/aspeed_g5_defconfig | 2 +-
> kernel/configs/debug.config | 2 +-
> kernel/hung_task.c | 15 ++++++++++-----
> lib/Kconfig.debug | 9 +++++----
> tools/testing/selftests/wireguard/qemu/kernel.config | 2 +-
> 7 files changed, 36 insertions(+), 23 deletions(-)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index a51ab46..492f0bc 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -1992,14 +1992,20 @@
> the added memory block itself do not be affected.
>
> hung_task_panic=
> - [KNL] Should the hung task detector generate panics.
> - Format: 0 | 1
> + [KNL] Number of hung tasks to trigger kernel panic.
> + Format: <int>
> +
> + When set to a non-zero value, a kernel panic will be triggered if
> + the number of detected hung tasks reaches this value.
> +
> + 0: don't panic
> + 1: panic immediately on first hung task
> + N: panic after N hung tasks are detected in a single scan
>
> - A value of 1 instructs the kernel to panic when a
> - hung task is detected. The default value is controlled
> - by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time
> - option. The value selected by this boot parameter can
> - be changed later by the kernel.hung_task_panic sysctl.
> + The default value is controlled by the
> + CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The value
> + selected by this boot parameter can be changed later by the
> + kernel.hung_task_panic sysctl.
>
> hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC)
> terminal devices. Valid values: 0..8
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index f3ee807..0065a55 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -397,13 +397,14 @@ a hung task is detected.
> hung_task_panic
> ===============
>
> -Controls the kernel's behavior when a hung task is detected.
> +When set to a non-zero value, a kernel panic will be triggered if the
> +number of hung tasks found during a single scan reaches this value.
> This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
>
> -= =================================================
> += =======================================================
> 0 Continue operation. This is the default behavior.
> -1 Panic immediately.
> -= =================================================
> +N Panic when N hung tasks are found during a single scan.
> += =======================================================
>
>
> hung_task_check_count
> diff --git a/arch/arm/configs/aspeed_g5_defconfig b/arch/arm/configs/aspeed_g5_defconfig
> index 61cee1e..c3b0d5f 100644
> --- a/arch/arm/configs/aspeed_g5_defconfig
> +++ b/arch/arm/configs/aspeed_g5_defconfig
> @@ -308,7 +308,7 @@ CONFIG_PANIC_ON_OOPS=y
> CONFIG_PANIC_TIMEOUT=-1
> CONFIG_SOFTLOCKUP_DETECTOR=y
> CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
> -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y
> +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1
> CONFIG_WQ_WATCHDOG=y
> # CONFIG_SCHED_DEBUG is not set
> CONFIG_FUNCTION_TRACER=y
> diff --git a/kernel/configs/debug.config b/kernel/configs/debug.config
> index e81327d..9f6ab7d 100644
> --- a/kernel/configs/debug.config
> +++ b/kernel/configs/debug.config
> @@ -83,7 +83,7 @@ CONFIG_SLUB_DEBUG_ON=y
> #
> # Debug Oops, Lockups and Hangs
> #
> -# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
> +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=0
> # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
> CONFIG_DEBUG_ATOMIC_SLEEP=y
> CONFIG_DETECT_HUNG_TASK=y
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index b2c1f14..84b4b04 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -81,7 +81,7 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace;
> * hung task is detected:
> */
> static unsigned int __read_mostly sysctl_hung_task_panic =
> - IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
> + CONFIG_BOOTPARAM_HUNG_TASK_PANIC;
>
> static int
> hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)
> @@ -218,8 +218,11 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
> }
> #endif
>
> -static void check_hung_task(struct task_struct *t, unsigned long timeout)
> +static void check_hung_task(struct task_struct *t, unsigned long timeout,
> + unsigned long prev_detect_count)
> {
> + unsigned long total_hung_task;
> +
> if (!task_is_hung(t, timeout))
> return;
>
> @@ -229,9 +232,10 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
> */
> sysctl_hung_task_detect_count++;
>
> + total_hung_task = sysctl_hung_task_detect_count - prev_detect_count;
> trace_sched_process_hang(t);
>
> - if (sysctl_hung_task_panic) {
> + if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) {
> console_verbose();
> hung_task_show_lock = true;
> hung_task_call_panic = true;
> @@ -300,6 +304,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> int max_count = sysctl_hung_task_check_count;
> unsigned long last_break = jiffies;
> struct task_struct *g, *t;
> + unsigned long prev_detect_count = sysctl_hung_task_detect_count;
>
> /*
> * If the system crashed already then all bets are off,
> @@ -320,7 +325,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> last_break = jiffies;
> }
>
> - check_hung_task(t, timeout);
> + check_hung_task(t, timeout, prev_detect_count);
> }
> unlock:
> rcu_read_unlock();
> @@ -389,7 +394,7 @@ static const struct ctl_table hung_task_sysctls[] = {
> .mode = 0644,
> .proc_handler = proc_dointvec_minmax,
> .extra1 = SYSCTL_ZERO,
> - .extra2 = SYSCTL_ONE,
> + .extra2 = SYSCTL_INT_MAX,
> },
> {
> .procname = "hung_task_check_count",
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 3034e294..3976c90 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -1258,12 +1258,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT
> Keeping the default should be fine in most cases.
>
> config BOOTPARAM_HUNG_TASK_PANIC
> - bool "Panic (Reboot) On Hung Tasks"
> + int "Number of hung tasks to trigger kernel panic"
> depends on DETECT_HUNG_TASK
> + default 0
> help
> - Say Y here to enable the kernel to panic on "hung tasks",
> - which are bugs that cause the kernel to leave a task stuck
> - in uninterruptible "D" state.
> + When set to a non-zero value, a kernel panic will be triggered
> + if the number of hung tasks found during a single scan reaches
> + this value.
>
> The panic can be used in combination with panic_timeout,
> to cause the system to reboot automatically after a
> diff --git a/tools/testing/selftests/wireguard/qemu/kernel.config b/tools/testing/selftests/wireguard/qemu/kernel.config
> index 936b18b..0504c11 100644
> --- a/tools/testing/selftests/wireguard/qemu/kernel.config
> +++ b/tools/testing/selftests/wireguard/qemu/kernel.config
> @@ -81,7 +81,7 @@ CONFIG_WQ_WATCHDOG=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
> CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
> -CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y
> +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1
> CONFIG_PANIC_TIMEOUT=-1
> CONFIG_STACKTRACE=y
> CONFIG_EARLY_PRINTK=y
> --
> 2.9.4
>
--
Masami Hiramatsu (Google) <mhiramat@...nel.org>
Powered by blists - more mailing lists