linux-kernel - Re: kernel deadlock

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <52261BBB.60904@linaro.org>
Date:	Tue, 03 Sep 2013 10:26:19 -0700
From:	John Stultz <john.stultz@...aro.org>
To:	Gerlando Falauto <gerlando.falauto@...mile.com>
CC:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Richard Cochran <richardcochran@...il.com>,
	Prarit Bhargava <prarit@...hat.com>,
	"Brunck, Holger" <Holger.Brunck@...mile.com>,
	"Longchamp, Valentin" <Valentin.Longchamp@...mile.com>,
	"Bigler, Stefan" <Stefan.Bigler@...mile.com>,
	Peter Zijlstra <peterz@...radead.org>
Subject: Re: kernel deadlock

On 09/03/2013 07:57 AM, Gerlando Falauto wrote:
> Hi,
>
> I tried again from scratch, so let me recap the whole situation, so we
> can all view it from the same standpoint. This should make the problem
> easier to see and reproduce.
>
> I can confirm that running a stock 3.10 kernel with HRTICK enabled:
[snip]
> makes the following program (and the whole board, as a matter of fact)
> hang with no further notice:
>
[snip]
> If I then revert everything up to (and including) the offending commit
> (mind the '~'):
>
> $ git log --oneline ...06c017f~ -- kernel/time/timekeeping.c
> kernel/time/ntp.c | cut -f1 -d' ' | xargs git revert
>
> The problem disappears.
>
> If I then cherry-pick again the offending commit:
>
> $ git cherry-pick 06c017f; git log -1
>
> commit 06c017fdd4dc48451a29ac37fc1db4a3f86b7f40
> Author: John Stultz <john.stultz@...aro.org>
> Date:   Fri Mar 22 11:37:28 2013 -0700
>
>      timekeeping: Hold timekeepering locks in do_adjtimex and hardpps
>
[snip]
> And as soon as I also cherry-pick (notice there is another commit in
> between, which seems not to be relevant on this matter):
>
> $ git cherry-pick a076b2146fabb0894cae5e0189a8ba3f1502d737; git show
>
> commit a076b2146fabb0894cae5e0189a8ba3f1502d737
> Author: John Stultz <john.stultz@...aro.org>
> Date:   Fri Mar 22 11:52:03 2013 -0700
>
>     ntp: Remove ntp_lock, using the timekeeping locks to protect ntp
> state
>
[snip]
>
> I end up in the situation where the system hangs completely and NO
> deadlock report whatsoever is output.
>
> So it looks like 06c017fdd4dc48451a29ac37fc1db4a3f86b7f40 introduces
> the deadlock, while a076b2146fabb0894cae5e0189a8ba3f1502d737 cares to
> hide the report.
>
> Notice how I tested the above on an ARM board; on PowerPC I get
> similar results, although I am not able to see the deadlock report
> under any circumstances (enabling CONFIG_PROVE_LOCKING, which is the
> flag that triggers the deadlock report, causes the kernel to hang at
> startup even on a vanilla 3.10 kernel).
>
> John, could you please confirm whether you're at least able to
> reproduce it somehow?

Thanks again for the detailed notes. I've so far been unable to
reproduce this, but I'm still looking at it.

Enabling the SCHED_FEAT(HRTICK, true) bit tends to cause lots of issues
on the various hardware I have, tripping the lockdep warnings on various
other issues:

[    4.224487] ======================================================
[    4.230987] [ INFO: possible circular locking dependency detected ]
[    4.237579] 3.9.0-rc2-00148-g06c017f-dirty #8 Not tainted
[    4.243255] -------------------------------------------------------
[    4.249847] kworker/0:1H/1329 is trying to acquire lock:
[    4.255432]  (&p->pi_lock){-.-.-.}, at: [<c006d184>]
try_to_wake_up+0x30/0x3c
[    4.263092]
[    4.263092] but task is already holding lock:
[    4.269226]  (&rq->lock){-.-.-.}, at: [<c0721f18>] __schedule+0xb8/0x7d0
[    4.276306]
[    4.276306] which lock already depends on the new lock.

and

[    2.360591] =============================================
[    2.363072] [ INFO: possible recursive locking detected ]
[    2.364882] 3.9.0-rc2+ #9 Not tainted
[    2.365486] ---------------------------------------------
[    2.366345] blkid/999 is trying to acquire lock:
[    2.367092]  (&p->pi_lock){-.-.-.}, at: [<ffffffff810c763c>]
try_to_wake_up+0x2c/0x300
[    2.367275]
[    2.367275] but task is already holding lock:
[    2.367275]  (&p->pi_lock){-.-.-.}, at: [<ffffffff810c763c>]
try_to_wake_up+0x2c/0x300
[    2.367275]
[    2.367275] other info that might help us debug this:
[    2.367275]  Possible unsafe locking scenario:
[    2.367275]

These warnings disable lockdep, so it may just be I'm not getting a
chance to see the warning you describe.


That said, the one system that doesn't throw those lockdep warnings,
doesn't seem to hit your lockdep warning either, and I've not been able
to trigger any hang using the test code you provided.

Just in case there's something else here at play, could you send me your
.config that you're using?

thanks
-john

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/