linux-kernel - Re: [syzbot] possible deadlock in worker

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 15 Feb 2022 19:26:18 +0900
From:   Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
To:     Tejun Heo <tj@...nel.org>
Cc:     Bart Van Assche <bvanassche@....org>,
        syzbot <syzbot+831661966588c802aae9@...kaller.appspotmail.com>,
        jgg@...pe.ca, linux-kernel@...r.kernel.org,
        linux-rdma@...r.kernel.org, syzkaller-bugs@...glegroups.com,
        Lai Jiangshan <jiangshanlai@...il.com>
Subject: Re: [syzbot] possible deadlock in worker_thread

On 2022/02/15 2:34, Tejun Heo wrote:
> 
> Instead of doing the above, please add a wq flag to mark system wqs and
> trigger the warning that way and I'd leave it regardless of PROVE_LOCKING.
> 

Do you mean something like below?
I think the above is easier to understand (for developers) because

  The above prints variable's name (one of 'system_wq', 'system_highpri_wq',
  'system_long_wq', 'system_unbound_wq', 'system_freezable_wq', 'system_power_efficient_wq'
  or 'system_freezable_power_efficient_wq') with backtrace (which will be translated into
  filename:line format) while the below prints queue's name (one of 'events', 'events_highpri',
  'events_long', 'events_unbound', 'events_freezable', 'events_power_efficient' or
  'events_freezable_power_efficient'). If CONFIG_KALLSYMS_ALL=y, we can print
  variable's name using "%ps", but CONFIG_KALLSYMS_ALL is not always enabled.

  The flag must not be used by user-defined WQs, for destroy_workqueue() involves
  flush_workqueue(). If this flag is by error used on user-defined WQs, pointless
  warning will be printed. I didn't pass this flag when creating system-wide WQs
  because some developer might copy&paste the
    system_wq = alloc_workqueue("events", 0, 0);
  lines when converting.

---
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 24 ++++++++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index 7fee9b6cfede..9e33dcd417d3 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -339,6 +339,7 @@ enum {
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
+	__WQ_SYSTEM_WIDE        = 1 << 20, /* interbal: don't flush this workqueue */
 
 	WQ_MAX_ACTIVE		= 512,	  /* I like 512, better ideas? */
 	WQ_MAX_UNBOUND_PER_CPU	= 4,	  /* 4 * #cpus for unbound wq */
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 33f1106b4f99..dbb9c6bb54a7 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -2805,6 +2805,21 @@ static bool flush_workqueue_prep_pwqs(struct workqueue_struct *wq,
 	return wait;
 }
 
+static void warn_if_flushing_global_workqueue(struct workqueue_struct *wq)
+{
+	static DEFINE_RATELIMIT_STATE(flush_warn_rs, 600 * HZ, 1);
+
+	if (likely(!(wq->flags & __WQ_SYSTEM_WIDE)))
+		return;
+
+	ratelimit_set_flags(&flush_warn_rs, RATELIMIT_MSG_ON_RELEASE);
+	if (!__ratelimit(&flush_warn_rs))
+		return;
+	pr_warn("Since system-wide WQ is shared, flushing system-wide WQ can introduce unexpected locking dependency. Please replace %s usage in your code with your local WQ.\n",
+		wq->name);
+	dump_stack();
+}
+
 /**
  * flush_workqueue - ensure that any scheduled work has run to completion.
  * @wq: workqueue to flush
@@ -2824,6 +2839,8 @@ void flush_workqueue(struct workqueue_struct *wq)
 	if (WARN_ON(!wq_online))
 		return;
 
+	warn_if_flushing_global_workqueue(wq);
+
 	lock_map_acquire(&wq->lockdep_map);
 	lock_map_release(&wq->lockdep_map);
 
@@ -6070,6 +6087,13 @@ void __init workqueue_init_early(void)
 	       !system_unbound_wq || !system_freezable_wq ||
 	       !system_power_efficient_wq ||
 	       !system_freezable_power_efficient_wq);
+	system_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_highpri_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_long_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_unbound_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_freezable_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_power_efficient_wq->flags |= __WQ_SYSTEM_WIDE;
+	system_freezable_power_efficient_wq->flags |= __WQ_SYSTEM_WIDE;
 }
 
 /**
-- 
2.32.0