| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <2f26d112e8834d378dd00f20cc384f39@baidu.com>
Date: Sat, 11 Oct 2025 12:03:23 +0000
From: "Li,Rongqing" <lirongqing@...du.com>
To: Masami Hiramatsu <mhiramat@...nel.org>
CC: "corbet@....net" <corbet@....net>, "akpm@...ux-foundation.org"
<akpm@...ux-foundation.org>, "lance.yang@...ux.dev" <lance.yang@...ux.dev>,
"paulmck@...nel.org" <paulmck@...nel.org>,
"pawan.kumar.gupta@...ux.intel.com" <pawan.kumar.gupta@...ux.intel.com>,
"mingo@...nel.org" <mingo@...nel.org>, "dave.hansen@...ux.intel.com"
<dave.hansen@...ux.intel.com>, "rostedt@...dmis.org" <rostedt@...dmis.org>,
"kees@...nel.org" <kees@...nel.org>, "arnd@...db.de" <arnd@...db.de>,
"feng.tang@...ux.alibaba.com" <feng.tang@...ux.alibaba.com>,
"pauld@...hat.com" <pauld@...hat.com>, "joel.granados@...nel.org"
<joel.granados@...nel.org>, "linux-doc@...r.kernel.org"
<linux-doc@...r.kernel.org>, "linux-kernel@...r.kernel.org"
<linux-kernel@...r.kernel.org>
Subject: RE: [????] Re: [PATCH][v2] hung_task: Panic after fixed number of
hung tasks
> -----Original Message-----
> From: Masami Hiramatsu <mhiramat@...nel.org>
> Sent: 2025年9月29日 8:48
> To: Li,Rongqing <lirongqing@...du.com>
> Cc: corbet@....net; akpm@...ux-foundation.org; lance.yang@...ux.dev;
> paulmck@...nel.org; pawan.kumar.gupta@...ux.intel.com; mingo@...nel.org;
> dave.hansen@...ux.intel.com; rostedt@...dmis.org; kees@...nel.org;
> arnd@...db.de; feng.tang@...ux.alibaba.com; pauld@...hat.com;
> joel.granados@...nel.org; linux-doc@...r.kernel.org;
> linux-kernel@...r.kernel.org
> Subject: [????] Re: [PATCH][v2] hung_task: Panic after fixed number of hung
> tasks
>
> On Sun, 28 Sep 2025 13:31:37 +0800
> lirongqing <lirongqing@...du.com> wrote:
>
> > From: Li RongQing <lirongqing@...du.com>
> >
> > Currently, when hung_task_panic is enabled, kernel will panic
> > immediately upon detecting the first hung task. However, some hung
> > tasks are transient and the system can recover fully, while others are
> > unrecoverable and trigger consecutive hung task reports, and a panic is
> expected.
> >
> > This commit adds a new sysctl parameter hung_task_count_to_panic to
> > allows specifying the number of consecutive hung tasks that must be
> > detected before triggering a kernel panic. This provides finer control
> > for environments where transient hangs maybe happen but persistent
> > hangs should still be fatal.
>
> IIUC, perhaps there are multiple groups that require different timeouts for
> hang checks, and you want to set the hung task timeout to match the shorter
> one, but ignore the longer ones at that point.
>
> If so, this is essentially a problem with a long process that is performed under
> TASK_UNINTERRUPTIBLE. Ideally, the progress of such process should be
> checked periodically and the hang check should be reset unless it is real
> blocked.
> But this is not currently implemented. (For example, depending on the media,
> it may not be possible to check whether long IO is being
> performed.)
>
> The hung_tasks will even simulate these types of hangs as task hang-ups. But if
> you set a long detection time accordingly, you will also have to wait until that
> detection time for hangs that occur in a short period of time.
>
> The hung tasks on one major lock can spread in a domino effect.
> So setting a reasonably short detection time, but not panicking until there are
> enough of them, seems like a reasonable strategy.
> But in this case, I think we also need a "hard timeout limit"
> of hung tasks, which will detect longer ones. And also you should use peak
> value not accumulation value.
>
> If it is really transient (thus, it is not hung), accumulation of such normal but
> just slow operation will still kick hung_tasks.
>
Is it reasonable to detect the existence of a hung task continuously for a certain number of times to trigger panic?
Like
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d17cd3f..045bef5 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -304,6 +304,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
int max_count = sysctl_hung_task_check_count;
unsigned long last_break = jiffies;
struct task_struct *g, *t;
+ unsigned long pre_detect_count = sysctl_hung_task_detect_count;
+ static unsigned long contiguous_detect_count;
/*
* If the system crashed already then all bets are off,
@@ -326,6 +328,15 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
check_hung_task(t, timeout);
}
+
+ if (sysctl_hung_task_detect_count != pre_detect_count) {
+ contiguous_detect_count++;
+ if (sysctl_max_hung_task_to_panic &&
+ contiguous_detect_count > sysctl_max_hung_task_to_panic)
+ hung_task_call_panic = 1;
+ }
+ else
+ contiguous_detect_count = 0;
unlock:
rcu_read_unlock();
if (hung_task_show_lock)
-Li
> Thank you,
>
> >
> > Acked-by: Lance Yang <lance.yang@...ux.dev>
> > Signed-off-by: Li RongQing <lirongqing@...du.com>
> > ---
> > Diff with v1: change documentation as Lance suggested
> >
> > Documentation/admin-guide/sysctl/kernel.rst | 8 ++++++++
> > kernel/hung_task.c | 14 +++++++++++++-
> > 2 files changed, 21 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/admin-guide/sysctl/kernel.rst
> > b/Documentation/admin-guide/sysctl/kernel.rst
> > index 8b49eab..98b47a7 100644
> > --- a/Documentation/admin-guide/sysctl/kernel.rst
> > +++ b/Documentation/admin-guide/sysctl/kernel.rst
> > @@ -405,6 +405,14 @@ This file shows up if
> ``CONFIG_DETECT_HUNG_TASK`` is enabled.
> > 1 Panic immediately.
> > = =================================================
> >
> > +hung_task_count_to_panic
> > +=====================
> > +
> > +When set to a non-zero value, a kernel panic will be triggered if the
> > +number of detected hung tasks reaches this value.
> > +
> > +Note that setting hung_task_panic=1 will still cause an immediate
> > +panic on the first hung task.
>
> What happen if it is 0?
>
> >
> > hung_task_check_count
> > =====================
> > diff --git a/kernel/hung_task.c b/kernel/hung_task.c index
> > 8708a12..87a6421 100644
> > --- a/kernel/hung_task.c
> > +++ b/kernel/hung_task.c
> > @@ -83,6 +83,8 @@ static unsigned int __read_mostly
> > sysctl_hung_task_all_cpu_backtrace;
> > static unsigned int __read_mostly sysctl_hung_task_panic =
> > IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC);
> >
> > +static unsigned int __read_mostly sysctl_hung_task_count_to_panic;
> > +
> > static int
> > hung_task_panic(struct notifier_block *this, unsigned long event,
> > void *ptr) { @@ -219,7 +221,9 @@ static void check_hung_task(struct
> > task_struct *t, unsigned long timeout)
> >
> > trace_sched_process_hang(t);
> >
> > - if (sysctl_hung_task_panic) {
> > + if (sysctl_hung_task_panic ||
> > + (sysctl_hung_task_count_to_panic &&
> > + (sysctl_hung_task_detect_count >=
> > +sysctl_hung_task_count_to_panic))) {
> > console_verbose();
> > hung_task_show_lock = true;
> > hung_task_call_panic = true;
> > @@ -388,6 +392,14 @@ static const struct ctl_table hung_task_sysctls[] = {
> > .extra2 = SYSCTL_ONE,
> > },
> > {
> > + .procname = "hung_task_count_to_panic",
> > + .data = &sysctl_hung_task_count_to_panic,
> > + .maxlen = sizeof(int),
> > + .mode = 0644,
> > + .proc_handler = proc_dointvec_minmax,
> > + .extra1 = SYSCTL_ZERO,
> > + },
> > + {
> > .procname = "hung_task_check_count",
> > .data = &sysctl_hung_task_check_count,
> > .maxlen = sizeof(int),
> > --
> > 2.9.4
> >
>
>
> --
> Masami Hiramatsu (Google) <mhiramat@...nel.org>
Powered by blists - more mailing lists