linux-kernel - Re: "Dying CPU not properly vacated" splat

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20220426000328.GY4285@paulmck-ThinkPad-P17-Gen-1>
Date:   Mon, 25 Apr 2022 17:03:28 -0700
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Valentin Schneider <vschneid@...hat.com>
Cc:     linux-kernel@...r.kernel.org, mingo@...hat.com,
        peterz@...radead.org, juri.lelli@...hat.com,
        vincent.guittot@...aro.org, dietmar.eggemann@....com,
        rostedt@...dmis.org, bsegall@...gle.com, mgorman@...e.de,
        bristot@...hat.com
Subject: Re: "Dying CPU not properly vacated" splat

On Mon, Apr 25, 2022 at 10:59:44PM +0100, Valentin Schneider wrote:
> On 25/04/22 10:33, Paul E. McKenney wrote:
> > On Mon, Apr 25, 2022 at 05:15:13PM +0100, Valentin Schneider wrote:
> >>
> >> Hi Paul,
> >>
> >> On 21/04/22 12:38, Paul E. McKenney wrote:
> >> > Hello!
> >> >
> >> > The rcutorture TREE03 scenario got the following splat, which appears
> >> > to be a one-off, or if not, having an MTBF in the thousands of hours,
> >> > even assuming that it is specific to TREE03.  (If it is not specific to
> >> > TREE03, we are talking tens of thousands of hours of rcutorture runtime.)
> >> >
> >> > So just in case this rings any bells or there are some diagnostics I
> >> > should add in case this ever happens again.  ;-)
> >>
> >> There should be a dump of the enqueued tasks right after the snippet you've
> >> sent, any chance you could share that if it's there? That should tell us
> >> which tasks are potentially misbehaving.
> >
> > And now that I know to look for them, there they are!  Thank you!!!
> >
> > CPU7 enqueued tasks (2 total):
> >  pid: 52, name: migration/7
> >  pid: 135, name: rcu_torture_rea
> > smpboot: CPU 7 is now offline
> >
> > So what did rcu_torture_reader() do wrong here?  ;-)
> >
> 
> So on teardown, CPUHP_AP_SCHED_WAIT_EMPTY->sched_cpu_wait_empty() waits for
> the rq to be empty. Tasks must *not* be enqueued onto that CPU after that
> step has been run - if there are per-CPU tasks bound to that CPU, they must
> be unbound in their respective hotplug callback.
> 
> For instance for workqueue.c, we have workqueue_offline_cpu() as a hotplug
> callback which invokes unbind_workers(cpu), the interesting bit being:
> 
>                 for_each_pool_worker(worker, pool) {
>                         kthread_set_per_cpu(worker->task, -1);
>                         WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
>                 }
> 
> The rcu_torture_reader() kthreads aren't bound to any particular CPU are
> they? I can't find any code that would indicate they are - and in that case
> it means we have a problem with is_cpu_allowed() or related.

I did not intend that the rcu_torture_reader() kthreads be bound, and
I am not seeing anything that binds them.

Thoughts?  (Other than that validating any alleged fix will be quite
"interesting".)

							Thanx, Paul