[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAJhGHyAso+JH+QMXcfKTyOqq4yBD9Vc19sBbebEXohHe5znL8Q@mail.gmail.com>
Date: Thu, 21 Jul 2022 11:35:52 +0800
From: Lai Jiangshan <jiangshanlai@...il.com>
To: Tejun Heo <tj@...nel.org>
Cc: Valentin Schneider <vschneid@...hat.com>,
LKML <linux-kernel@...r.kernel.org>,
Peter Zijlstra <peterz@...radead.org>,
Frederic Weisbecker <frederic@...nel.org>,
Juri Lelli <juri.lelli@...hat.com>,
Phil Auld <pauld@...hat.com>,
Marcelo Tosatti <mtosatti@...hat.com>
Subject: Re: [RFC PATCH] workqueue: Unbind workers before sending them to exit()
> +static void unbind_worker(struct worker *worker)
> +{
> + kthread_set_per_cpu(worker->task, -1);
> + WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, wq_unbound_cpumask) < 0);
> +}
> +
> +static void rebind_worker(struct worker *worker, struct worker_pool *pool)
> +{
> + kthread_set_per_cpu(worker->task, pool->cpu);
> + WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask) < 0);
> +}
> +
> /**
> * destroy_worker - destroy a workqueue worker
> * @worker: worker to be destroyed
> @@ -1999,6 +2011,16 @@ static void destroy_worker(struct worker *worker)
>
> list_del_init(&worker->entry);
> worker->flags |= WORKER_DIE;
> +
> + /*
> + * We're sending that thread off to die, so any CPU would do. This is
> + * especially relevant for pcpu kworkers affined to an isolated CPU:
> + * we'd rather not interrupt an isolated CPU just for a kworker to
> + * do_exit().
> + */
> + if (!(worker->flags & WORKER_UNBOUND))
> + unbind_worker(worker);
> +
> wake_up_process(worker->task);
> }
destroy_worker() is called with raw_spin_lock_irq(pool->lock), so
it cannot call the sleepable set_cpus_allowed_ptr().
>From __set_cpus_allowed_ptr:
> * NOTE: the caller must have a valid reference to the task, the
> * task must not exit() & deallocate itself prematurely. The
> * call is not atomic; no spinlocks may be held.
I think it needs something like task_set_cpumask_possible() which is
documented as being usable in (raw) spinlocks and set the task's cpumask
to cpu_possible_mask and let the later ttwu help migrate it to a
proper non-isolated CPU or let it keep running.
On Thu, Jul 21, 2022 at 2:03 AM Tejun Heo <tj@...nel.org> wrote:
>
> On Tue, Jul 19, 2022 at 05:57:43PM +0100, Valentin Schneider wrote:
> > It has been reported that isolated CPUs can suffer from interference due to
> > per-CPU kworkers waking up just to die.
> >
> > A surge of workqueue activity (sleeping workfn's exacerbate this) during
> > initial setup can cause extra per-CPU kworkers to be spawned. Then, a
> > latency-sensitive task can be running merrily on an isolated CPU only to be
> > interrupted sometime later by a kworker marked for death (cf.
> > IDLE_WORKER_TIMEOUT, 5 minutes after last kworker activity).
> >
> > Affine kworkers to the wq_unbound_cpumask (which doesn't contain isolated
> > CPUs, cf. HK_TYPE_WQ) before waking them up after marking them with
> > WORKER_DIE.
> >
> > This follows the logic of CPU hot-unplug, which has been packaged into
> > helpers for the occasion.
>
> Idea-wise, seems fine to me, but we have some other issues around twiddling
> cpu affinities right now, so let's wait a bit till Lai chimes in.
>
I think there are some imperfections related to cpu affinities
which need to be fixed too.
Thanks
Lai
Powered by blists - more mailing lists