linux-kernel - Re: [BUG] Boot hangs at clocksource_done

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 1 Sep 2015 22:54:27 +0200 (CEST)
From:	Thomas Gleixner <tglx@...utronix.de>
To:	Alex Thorlton <athorlton@....com>
cc:	linux-kernel@...r.kernel.org,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>,
	John Stultz <john.stultz@...aro.org>,
	Russ Anderson <rja@....com>,
	Dimitri Sivanich <sivanich@....com>
Subject: Re: [BUG] Boot hangs at clocksource_done_booting on large configs

On Tue, 1 Sep 2015, Alex Thorlton wrote:
> On Mon, Aug 31, 2015 at 11:12:12PM +0200, Thomas Gleixner wrote:
> > But they are obviously held off by something. And that something looks
> > like the timekeeper sequence lock. 
> 
> Ok.  IIRC, that lock prevents reads to the clocksource until all
> necessary write operations have finished, correct?  i.e. tk_core.seq?
> 
> I looked at that as being a possible place to get stuck in ktime_get:
> 
>         do {
>                 seq = read_seqcount_begin(&tk_core.seq);
>                 base = tk->tkr_mono.base;
>                 nsecs = timekeeping_get_ns(&tk->tkr_mono);
> 
>         } while (read_seqcount_retry(&tk_core.seq, seq));
> 
> But I don't have enough evidence to point there for sure.

Right and it might be a red herring. If the access to HPET takes long
enough for each CPU then you get into a situation where you never get
out of those timer interrupts anymore.
 
> > Your patch shifts some timing around so the issue does not happen, but
> > that's certainly not a solution.
> 
> I was fairly sure of that, but it's good to know that you agree.
> 
> I'll try and get a backtrace from a completely unmodified kernel here in
> the next few days.  Unfortunately, as mentioned, we only hit the issue
> intermittently.  I was able to reproduce pretty regularly on kernels
> from 4.0 and before, but it seems to be less common on newer kernels,
> though we definitely still manage to hit it from time to time.

Well, I doubt that backtraces will tell us much. Some moderate form of
tracing (a few tracepoints enabled) and a trace_dump(DUMP_ALL); call
before or instead of the NMI backtrace stuff might give us the really
interesting information. i.e. what the cpus have been doing for the
last second or such. The backtraces are just momentary snapshots and
not necessarily helpful.

You might have to hack the trace dump code to limit the output to a
few cpus if you don't want to wait a day or two until the information
has spilled out over serial :)

Thanks,

	tglx

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/