linux-kernel - Re: [PATCH v10 1/3] smpboot: allow excluding cpus from the smpboot threads

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150501212329.GA4179@lerouge>
Date:	Fri, 1 May 2015 23:23:30 +0200
From:	Frederic Weisbecker <fweisbec@...il.com>
To:	Chris Metcalf <cmetcalf@...hip.com>
Cc:	Andrew Morton <akpm@...ux-foundation.org>,
	Don Zickus <dzickus@...hat.com>,
	Ingo Molnar <mingo@...nel.org>,
	Andrew Jones <drjones@...hat.com>,
	Ulrich Obergfell <uobergfe@...hat.com>,
	Fabian Frederick <fabf@...net.be>,
	Aaron Tomlin <atomlin@...hat.com>,
	Ben Zhang <benzh@...omium.org>,
	Christoph Lameter <cl@...ux.com>,
	Gilad Ben-Yossef <gilad@...yossef.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	linux-kernel@...r.kernel.org, Jonathan Corbet <corbet@....net>,
	linux-doc@...r.kernel.org, Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>
Subject: Re: [PATCH v10 1/3] smpboot: allow excluding cpus from the smpboot
 threads

On Fri, May 01, 2015 at 03:57:51PM -0400, Chris Metcalf wrote:
> On 05/01/2015 04:53 AM, Frederic Weisbecker wrote:
> >>+	/* Unpark any threads that were voluntarily parked. */
> >>+	for_each_cpu_not(cpu, &ht->cpumask) {
> >>+		if (cpu_online(cpu)) {
> >>+			struct task_struct *tsk = *per_cpu_ptr(ht->store, cpu);
> >>+			if (tsk)
> >>+				kthread_unpark(tsk);
> >I'm still not clear why we are doing that. kthread_stop() should be able
> >to handle parked kthreads, otherwise it needs to be fixed.
> 
> Checking without the unpark, it's actually only a problem with nohz_full.
> In a system without nohz_full, the kthreads are able to stop even when
> they are parked; it's only in the nohz_full case that things wedge.

Ok. So this isn't a proper fix but a workaround for a bug that we don't
understand yet. In this case I much prefer that you remove this workaround
(I'm talking about this unpark loop) because it hides the issue. And hiding
the bug is the last thing we want if we plan to fix it properly.

> 
> For example, booting with only cpu 0 as a housekeeping core (and
> therefore all watchdogs 1-35 on my 36-core tilegx are parked), and
> immediately doing "echo 0 > /proc/sys/kernel/watchdog", I see
> (via SysRq ^O-l) the first parked watchdog, on cpu 1, hung with:
> 
>   frame 0: 0xfffffff7000f2928 lock_hrtimer_base+0xb8/0xc0
>   frame 1: 0xfffffff7000f2a28 hrtimer_try_to_cancel+0x40/0x170
>   frame 2: 0xfffffff7000f2a28 hrtimer_try_to_cancel+0x40/0x170
>   frame 3: 0xfffffff7000f2b98 hrtimer_cancel+0x40/0x68
>   frame 4: 0xfffffff70014cce0 watchdog_disable+0x50/0x70
>   frame 5: 0xfffffff70008c2d0 smpboot_thread_fn+0x350/0x438
>   frame 6: 0xfffffff700084b28 kthread+0x160/0x178

Have you tried to do that before your patchset?

> The other cores are all idle.
> 
> I have no idea why lock_hrtimer_base() is hanging; perhaps the
> hrtimer_cpu_base lock is taken by some other task that is now
> scheduled out.

No, it's a spinlock, tasks can't sleep while holding it. But it looks like
a deadlock.

> 
> The config does not have NO_HZ_FULL_ALL or NO_HZ_FULL_SYSIDLE
> set, and does have RCU_FAST_NO_HZ and RCU_NOCB_CPU_ALL.
> 
> I don't really know how to start debugging this, but I do know that
> unparking the threads first avoids the issue :-)

Do you have CONFIG_PROVE_LOCKING=y ?

I can't check that myself until the middle of next week.

> 
> -- 
> Chris Metcalf, EZChip Semiconductor
> http://www.ezchip.com
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/