[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-Id: <20230201134543.13687-1-pmladek@suse.com>
Date: Wed, 1 Feb 2023 14:45:38 +0100
From: Petr Mladek <pmladek@...e.com>
To: Tejun Heo <tj@...nel.org>
Cc: Lai Jiangshan <jiangshanlai@...il.com>,
Michal Koutny <mkoutny@...e.com>, linux-kernel@...r.kernel.org,
Petr Mladek <pmladek@...e.com>
Subject: [RFC 0/5] workqueue: Debugging improvements
The workqueue watchdog provides a lot of impormation when a stall is
detected. The report says a lot about what workqueues and worker pools
are active and what is being blocked. Unfortunately, it does not provide
much information about what caused the stall.
In particular, it did not help me to get root of the following problems:
+ New workers were not created because the system reached PID limit.
Admins limited it too much in a cloud.
+ A networking driver was not loaded because systemd killed modprobe
when switching the root from initrd to the booted system.
It was surprisingly quite reproducible. Interrupts are not handled
immediately in kernel code. The wait in kthread_create_on_node()
was one of few locations. So the race window evidently was not
trivial.
1st patch fixes a misleading "hung" time report.
2nd, 3rd, and 4rd patches add warnings into create_worker() and
create_rescuer(). The rather persistent errors are printed only once
until it succeeds again. Otherwise it would be too noisy and might even
break the watchdog.
5th patch adds printing bracktraces of CPU-bound workers that might
block CPU-bound workqueues. The candidate is well defined to keep
the number of backtraces small. It always printed only the right one
during my testing.
The first 4 patches would have helped me to debug the real problems
that I met.
The 5th patch is theoretical. I did not see this case in practice.
But it looks realistic enough. And it worked very well when I
simulated the problem. IMHO, it should be pretty useful.
Petr Mladek (5):
workqueue: Fix hung time report of worker pools
workqueue: Warn when a new worker could not be created
workqueue: Interrupted create_worker() is not a repeated event
workqueue: Warn when a rescuer could not be created
workqueue: Print backtraces from CPUs with hung CPU bound workqueues
kernel/workqueue.c | 186 ++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 177 insertions(+), 9 deletions(-)
--
2.35.3
Powered by blists - more mailing lists