linux-kernel - Re: [syzbot] possible deadlock in worker

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <76616D2F-14F2-4D83-9DB4-576FB2ACB72C@oracle.com>
Date:   Tue, 15 Feb 2022 10:43:47 +0000
From:   Haakon Bugge <haakon.bugge@...cle.com>
To:     Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
CC:     Tejun Heo <tj@...nel.org>, Bart Van Assche <bvanassche@....org>,
        syzbot <syzbot+831661966588c802aae9@...kaller.appspotmail.com>,
        Jason Gunthorpe <jgg@...pe.ca>,
        LKML <linux-kernel@...r.kernel.org>,
        OFED mailing list <linux-rdma@...r.kernel.org>,
        "syzkaller-bugs@...glegroups.com" <syzkaller-bugs@...glegroups.com>,
        Lai Jiangshan <jiangshanlai@...il.com>
Subject: Re: [syzbot] possible deadlock in worker_thread



> On 15 Feb 2022, at 11:26, Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp> wrote:
> 
> On 2022/02/15 2:34, Tejun Heo wrote:
>> 
>> Instead of doing the above, please add a wq flag to mark system wqs and
>> trigger the warning that way and I'd leave it regardless of PROVE_LOCKING.
>> 
> 
> Do you mean something like below?
> I think the above is easier to understand (for developers) because
> 
>  The above prints variable's name (one of 'system_wq', 'system_highpri_wq',
>  'system_long_wq', 'system_unbound_wq', 'system_freezable_wq', 'system_power_efficient_wq'
>  or 'system_freezable_power_efficient_wq') with backtrace (which will be translated into
>  filename:line format) while the below prints queue's name (one of 'events', 'events_highpri',
>  'events_long', 'events_unbound', 'events_freezable', 'events_power_efficient' or
>  'events_freezable_power_efficient'). If CONFIG_KALLSYMS_ALL=y, we can print
>  variable's name using "%ps", but CONFIG_KALLSYMS_ALL is not always enabled.
> 
>  The flag must not be used by user-defined WQs, for destroy_workqueue() involves
>  flush_workqueue(). If this flag is by error used on user-defined WQs, pointless
>  warning will be printed. I didn't pass this flag when creating system-wide WQs
>  because some developer might copy&paste the
>    system_wq = alloc_workqueue("events", 0, 0);
>  lines when converting.
> 
> ---
> include/linux/workqueue.h |  1 +
> kernel/workqueue.c        | 24 ++++++++++++++++++++++++
> 2 files changed, 25 insertions(+)
> 
> diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
> index 7fee9b6cfede..9e33dcd417d3 100644
> --- a/include/linux/workqueue.h
> +++ b/include/linux/workqueue.h
> @@ -339,6 +339,7 @@ enum {
> 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
> 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
> 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
> +	__WQ_SYSTEM_WIDE        = 1 << 20, /* interbal: don't flush this workqueue */

s/interbal/internal/

> 
> 	WQ_MAX_ACTIVE		= 512,	  /* I like 512, better ideas? */
> 	WQ_MAX_UNBOUND_PER_CPU	= 4,	  /* 4 * #cpus for unbound wq */
> diff --git a/kernel/workqueue.c b/kernel/workqueue.c
> index 33f1106b4f99..dbb9c6bb54a7 100644
> --- a/kernel/workqueue.c
> +++ b/kernel/workqueue.c
> @@ -2805,6 +2805,21 @@ static bool flush_workqueue_prep_pwqs(struct workqueue_struct *wq,
> 	return wait;
> }
> 
> +static void warn_if_flushing_global_workqueue(struct workqueue_struct *wq)
> +{
> +	static DEFINE_RATELIMIT_STATE(flush_warn_rs, 600 * HZ, 1);
> +
> +	if (likely(!(wq->flags & __WQ_SYSTEM_WIDE)))
> +		return;
> +
> +	ratelimit_set_flags(&flush_warn_rs, RATELIMIT_MSG_ON_RELEASE);
> +	if (!__ratelimit(&flush_warn_rs))
> +		return;
> +	pr_warn("Since system-wide WQ is shared, flushing system-wide WQ can introduce unexpected locking dependency. Please replace %s usage in your code with your local WQ.\n",
> +		wq->name);
> +	dump_stack();
> +}
> +
> /**
>  * flush_workqueue - ensure that any scheduled work has run to completion.
>  * @wq: workqueue to flush
> @@ -2824,6 +2839,8 @@ void flush_workqueue(struct workqueue_struct *wq)
> 	if (WARN_ON(!wq_online))
> 		return;
> 
> +	warn_if_flushing_global_workqueue(wq);
> +
> 	lock_map_acquire(&wq->lockdep_map);
> 	lock_map_release(&wq->lockdep_map);
> 
> @@ -6070,6 +6087,13 @@ void __init workqueue_init_early(void)
> 	       !system_unbound_wq || !system_freezable_wq ||
> 	       !system_power_efficient_wq ||
> 	       !system_freezable_power_efficient_wq);
> +	system_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_highpri_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_long_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_unbound_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_freezable_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_power_efficient_wq->flags |= __WQ_SYSTEM_WIDE;
> +	system_freezable_power_efficient_wq->flags |= __WQ_SYSTEM_WIDE;

Better to OR this in, in the alloc_workqueue() call? Perceive the notion of an opaque object?


Thxs, Håkon

> }
> 
> /**
> -- 
> 2.32.0
> 
>