linux-kernel - Re: [PATCH RT 10/30] hrtimer: Prevent using hrtimer_grab_expiry_lock() on migration

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ac4168c7-68d6-e558-644e-e0daceea1f61@prevas.dk>
Date:   Mon, 27 Apr 2020 15:10:00 +0200
From:   Rasmus Villemoes <rasmus.villemoes@...vas.dk>
To:     Steven Rostedt <rostedt@...dmis.org>, linux-kernel@...r.kernel.org,
        linux-rt-users <linux-rt-users@...r.kernel.org>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        Carsten Emde <C.Emde@...dl.org>,
        Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        John Kacur <jkacur@...hat.com>,
        Julia Cartwright <julia@...com>,
        Daniel Wagner <wagi@...om.org>,
        Tom Zanussi <zanussi@...nel.org>,
        Julien Grall <julien.grall@....com>
Subject: Re: [PATCH RT 10/30] hrtimer: Prevent using
 hrtimer_grab_expiry_lock() on migration_base

On 23/01/2020 21.39, Steven Rostedt wrote:
> 4.19.94-rt39-rc2 stable review patch.
> If anyone has any objections, please let me know.
> 
> ------------------
> 
> From: Julien Grall <julien.grall@....com>
> 
> [ Upstream commit cef1b87f98823af923a386f3f69149acb212d4a1 ]
> 
> As tglx puts it:
> |If base == migration_base then there is no point to lock soft_expiry_lock
> |simply because the timer is not executing the callback in soft irq context
> |and the whole lock/unlock dance can be avoided.
> 
> Furthermore, all the path leading to hrtimer_grab_expiry_lock() assumes
> timer->base and timer->base->cpu_base are always non-NULL. So it is safe
> to remove the NULL checks here.
> 
> Signed-off-by: Julien Grall <julien.grall@....com>
> Link: https://lkml.kernel.org/r/alpine.DEB.2.21.1908211557420.2223@nanos.tec.linutronix.de
> Signed-off-by: Steven Rostedt (VMware) <rostedt@...dmis.org>
> [bigeasy: rewrite changelog]
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@...utronix.de>

Pretty late to the party, but I think I've bisected a problem to this
patch (and its required fixup for !SMP, "hrtimer: Add a missing bracket
and hide `migration_base on !SMP").

Originally, a customer reported that upgrading from 4.19.82-rt30 to
v4.19.106-rt45 failed to boot, stalling around the time the network gets
initialized (this is a board with an embedded Marvell switch). Perhaps 1
in 10 times, the board would come up successfully. I haven't been able
to reproduce that particular problem (or, perhaps I've seen it once or
twice, but not nearly often enough to use that as a basis for bisection).

However, building with their rescue initrd and booting that, the board
would consistently hang during reboot. Sometimes I would get lines like

[   72.956630] sched: RT throttling activated
[   72.973769] lanx: port 1(lan1) entered disabled state
[   73.000401] lanx: port 2(lan2) entered disabled state
[   73.974951] lanx: port 3(lan3) entered disabled state
[   73.997473] lanx: port 4(lan4) entered disabled state
[   74.968006] lanx: port 5(lan5) entered disabled state

other times there would be no output, but the board was still hanging.
Reverting

b1a471ec4df1 - hrtimer: Prevent using hrtimer_grab_expiry_lock() on
migration_base
40aae5708e7a - hrtimer: Add a missing bracket and hide `migration_base'
on !SMP

on top of v4.19.94-rt39 makes that problem go away, i.e. the board
reboots as expected.

The board is a 32 bit powerpc (mpc8309) !SMP. Any ideas what I can do to
debug this further?

Thanks,
Rasmus