linux-kernel - Re: [PATCH resend] hung_task: add task->flags, blocked by coredump to log

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-Id: <20250110195445.586a1682302e66788e83e83e@linux-foundation.org>
Date: Fri, 10 Jan 2025 19:54:45 -0800
From: Andrew Morton <akpm@...ux-foundation.org>
To: Oxana Kharitonova <oxana@...udflare.com>
Cc: brauner@...nel.org, bsegall@...gle.com, dietmar.eggemann@....com,
 jack@...e.cz, juri.lelli@...hat.com, mgorman@...e.de, mingo@...hat.com,
 peterz@...radead.org, rostedt@...dmis.org, vincent.guittot@...aro.org,
 viro@...iv.linux.org.uk, vschneid@...hat.com, kernel-team@...udflare.com,
 linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH resend] hung_task: add task->flags, blocked by coredump
 to log

On Fri, 10 Jan 2025 16:03:28 +0000 Oxana Kharitonova <oxana@...udflare.com> wrote:

> Resending this patch as I haven't received feedback on my initial 
> submission https://lore.kernel.org/all/20241204182953.10854-1-oxana@cloudflare.com/
> 
> For the processes which are terminated abnormally the kernel can provide 
> a coredump if enabled. When the coredump is performed, the process and 
> all its threads are put into the D state 
> (TASK_UNINTERRUPTIBLE | TASK_FREEZABLE). 
> 
> On the other hand, we have kernel thread khungtaskd which monitors the 
> processes in the D state. If the task stuck in the D state more than 
> kernel.hung_task_timeout_secs, the hung_task alert appears in the kernel 
> log.
> 
> The higher memory usage of a process, the longer it takes to create 
> coredump, the longer tasks are in the D state. We have hung_task alerts 
> for the processes with memory usage above 10Gb. Although, our 
> kernel.hung_task_timeout_secs is 10 sec when the default is 120 sec.
> 
> Adding additional information to the log that the task is blocked by 
> coredump will help with monitoring. Another approach might be to 
> completely filter out alerts for such tasks, but in that case we would 
> lose transparency about what is putting pressure on some system 
> resources, e.g. we saw an increase in I/O when coredump occurs due its 
> writing to disk.
> 
> Additionally, it would be helpful to have task_struct->flags in the log 
> from the function sched_show_task(). Currently it prints 
> task_struct->thread_info->flags, this seems misleading as the line 
> starts with "task:xxxx".
> 

I added the below fix.


From: Andrew Morton <akpm@...ux-foundation.org>
Subject: hung_task-add-task-flags-blocked-by-coredump-to-log-fix
Date: Fri Jan 10 07:51:53 PM PST 2025

fix printk control string

In file included from ./include/asm-generic/bug.h:22,
                 from ./arch/x86/include/asm/bug.h:99,
                 from ./include/linux/bug.h:5,
                 from ./arch/x86/include/asm/paravirt.h:19,
                 from ./arch/x86/include/asm/irqflags.h:80,
                 from ./include/linux/irqflags.h:18,
                 from ./include/linux/spinlock.h:59,
                 from ./include/linux/wait.h:9,
                 from ./include/linux/wait_bit.h:8,
                 from ./include/linux/fs.h:6,
                 from ./include/linux/highmem.h:5,
                 from kernel/sched/core.c:10:
kernel/sched/core.c: In function 'sched_show_task':
./include/linux/kern_levels.h:5:25: error: format '%lx' expects argument of type 'long unsigned int', but argument 6 has type 'unsigned int' [-Werror=format=]
    5 | #define KERN_SOH        "\001"          /* ASCII Start Of Header */
      |                         ^~~~~~
./include/linux/printk.h:473:25: note: in definition of macro 'printk_index_wrap'
  473 |                 _p_func(_fmt, ##__VA_ARGS__);                           \
      |                         ^~~~
./include/linux/printk.h:586:9: note: in expansion of macro 'printk'
  586 |         printk(KERN_CONT fmt, ##__VA_ARGS__)
      |         ^~~~~~
./include/linux/kern_levels.h:24:25: note: in expansion of macro 'KERN_SOH'
   24 | #define KERN_CONT       KERN_SOH "c"
      |                         ^~~~~~~~
./include/linux/printk.h:586:16: note: in expansion of macro 'KERN_CONT'
  586 |         printk(KERN_CONT fmt, ##__VA_ARGS__)
      |                ^~~~~~~~~
kernel/sched/core.c:7704:9: note: in expansion of macro 'pr_cont'
 7704 |         pr_cont(" stack:%-5lu pid:%-5d tgid:%-5d ppid:%-6d task_flags:0x%08lx flags:0x%08lx\n",
      |         ^~~~~~~
cc1: all warnings being treated as errors

Cc: Al Viro <viro@...iv.linux.org.uk>
Cc: Ben Segall <bsegall@...gle.com>
Cc: Christian Brauner <brauner@...nel.org>
Cc: Dietmar Eggemann <dietmar.eggemann@....com>
Cc: Ingo Molnar <mingo@...hat.com>
Cc: Jan Kara <jack@...e.cz>
Cc: Juri Lelli <juri.lelli@...hat.com>
Cc: Mel Gorman <mgorman@...e.de>
Cc: Oxana Kharitonova <oxana@...udflare.com>
Cc: Peter Zijlstra (Intel) <peterz@...radead.org>
Cc: Steven Rostedt <rostedt@...dmis.org>
Cc: Valentin Schneider <vschneid@...hat.com>
Cc: Vincent Guittot <vincent.guittot@...aro.org>
Signed-off-by: Andrew Morton <akpm@...ux-foundation.org>
---

 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/kernel/sched/core.c~hung_task-add-task-flags-blocked-by-coredump-to-log-fix
+++ a/kernel/sched/core.c
@@ -7701,7 +7701,7 @@ void sched_show_task(struct task_struct
 	if (pid_alive(p))
 		ppid = task_pid_nr(rcu_dereference(p->real_parent));
 	rcu_read_unlock();
-	pr_cont(" stack:%-5lu pid:%-5d tgid:%-5d ppid:%-6d task_flags:0x%08lx flags:0x%08lx\n",
+	pr_cont(" stack:%-5lu pid:%-5d tgid:%-5d ppid:%-6d task_flags:0x%04x flags:0x%08lx\n",
 		free, task_pid_nr(p), task_tgid_nr(p),
 		ppid, p->flags, read_task_thread_flags(p));
 
_