lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20241027120747.42833-1-ioworker0@gmail.com>
Date: Sun, 27 Oct 2024 20:07:45 +0800
From: Lance Yang <ioworker0@...il.com>
To: akpm@...ux-foundation.org
Cc: dj456119@...il.com,
	cunhuang@...cent.com,
	leonylgao@...cent.com,
	j.granados@...sung.com,
	jsiddle@...hat.com,
	kent.overstreet@...ux.dev,
	21cnbao@...il.com,
	ryan.roberts@....com,
	david@...hat.com,
	ziy@...dia.com,
	libang.li@...group.com,
	baolin.wang@...ux.alibaba.com,
	linux-kernel@...r.kernel.org,
	linux-mm@...ck.org,
	joel.granados@...nel.org,
	linux@...ssschuh.net,
	Lance Yang <ioworker0@...il.com>
Subject: [PATCH v2 0/2] add detect count for hung tasks

Hi all,

This patchset adds a counter, hung_task_detect_count, to track the number
of times hung tasks are detected. 

IHMO, hung tasks are a critical metric. Currently, we detect them by
periodically parsing dmesg. However, this method isn't as user-friendly
as using a counter.

Sometimes, a short-lived issue with NIC or hard drive can quickly decrease
the hung_task_warnings to zero. Without warnings, we must directly access
the node to ensure that there are no more hung tasks and that the system
has recovered. After all, load average alone cannot provide a clear
picture.

Once this counter is in place, in a high-density deployment pattern, we
plan to set hung_task_timeout_secs to a lower number to improve stability,
even though this might result in false positives. And then we can set a
time-based threshold: if hung tasks last beyond this duration, we will
automatically migrate containers to other nodes. Based on past experience,
this approach could help avoid many production disruptions.

Moreover, just like other important events such as OOM that already have
counters, having a dedicated counter for hung tasks makes sense ;)

---

Changes since v1 [1]
====================
 - hung_task: add detect count for hung tasks
   - Update the changelog (per Andrew)
   - Find other folks to CC (per Andrew)

[1] https://lore.kernel.org/linux-mm/20241022114736.83285-1-ioworker0@gmail.com

Lance Yang (2):
  hung_task: add detect count for hung tasks
  hung_task: add docs for hung_task_detect_count

 Documentation/admin-guide/sysctl/kernel.rst |  9 +++++++++
 kernel/hung_task.c                          | 18 ++++++++++++++++++
 2 files changed, 27 insertions(+)

-- 
2.45.2


Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ