linux-kernel - Re: cpu stopper threads and load balancing leads to deadlock

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1524030475.5645.2.camel@gmx.de>
Date:   Wed, 18 Apr 2018 07:47:55 +0200
From:   Mike Galbraith <gleep@....de>
To:     Matt Fleming <matt@...eblueprint.co.uk>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...nel.org>
Cc:     linux-kernel@...r.kernel.org, Michal Hocko <mhocko@...e.com>
Subject: Re: cpu stopper threads and load balancing leads to deadlock

On Tue, 2018-04-17 at 15:21 +0100, Matt Fleming wrote:
> Hi guys,
> 
> We've seen a bug in one of our SLE kernels where the cpu stopper
> thread ("migration/15") is entering idle balance. This then triggers
> active load balance.
> 
> At the same time, a task on another CPU triggers a page fault and NUMA
> balancing kicks in to try and migrate the task closer to the NUMA node
> for that page (we're inside stop_two_cpus()). This faulting task is
> spinning in try_to_wake_up() (inside smp_cond_load_acquire(&p->on_cpu,
> !VAL)), waiting for "migration/15" to context switch.
> 
> Unfortunately, because "migration/15" is doing active load balance
> it's spinning waiting for the NUMA-page-faulting CPU's stopper lock,
> which is already held (since it's inside stop_two_cpus()).
> 
> Deadlock ensues.
> 
> This seems like a situation that should be prohibited, but I cannot
> find any code to prevent it. Is it OK for stopper threads to load
> balance? Is there something that should prevent this situation from
> happening?

I don't see anything to stop the deadlock either, would exclude stop
class from playing idle balancer entirely, though I suppose you could
check for caller being stop class in need_active_balance().  I don't
think any RT class playing idle balancer is particularly wonderful.

	-Mike