linux-kernel - Re: [PATCH] watchdog: Fix possible soft lockup warning at bootup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20200102120817.d1c289313747cfde7270076f@linux-foundation.org>
Date:   Thu, 2 Jan 2020 12:08:17 -0800
From:   Andrew Morton <akpm@...ux-foundation.org>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        linux-kernel@...r.kernel.org, Mike Rapoport <rppt@...ux.ibm.com>,
        Kees Cook <keescook@...omium.org>
Subject: Re: [PATCH] watchdog: Fix possible soft lockup warning at bootup

On Thu,  2 Jan 2020 10:41:49 -0500 Waiman Long <longman@...hat.com> wrote:

> It was found that watchdog soft lockup warning was displayed on some
> arm64 server systems at bootup time:
> 
> ...
>
> Further analysis of the situation revealed that the smp_init() call
> itself took more than 20s for that 2-socket 56-core and 224-thread
> server.
> 
>  [    0.115632] CPU1: Booted secondary processor 0x0000000100 [0x431f0af1]
>    :
>  [   27.177282] CPU223: Booted secondary processor 0x0000011b03 [0x431f0af1]
> 
> By adding some instrumentation code, it was found that for cpu 14,
> watchdog_enable() was called early with a timestamp of 1. The first
> watchdog timer callback for that cpu, however, happened really late at
> the above 25s timestamp mark causing the watchdog logic to treat the
> delay as a soft lockup.
> 
> On another arm64 system that doesn't show the soft lockup warning, the
> watchdog timer callback happened earlier at the 5s timestamp mark with
> the watchdog thread invoked shortly after that.
> 
> The reason why there was such a delay in the first watchdog timer
> callback for that particular system wasn't fully known yet.

Mysteries are unwelcome.  Are you continuing to investigate this?

> Given
> the fact that smp_init() can run for a long time on some systems,
> it is probably more appropriate to enable the watchdog function after
> smp_init() instead of before it.
> 
> Another way is to leave watchdog_touch_ts at 0 in watchdog_enable()
> while the system is at the booting stage. Either one of those should
> be able to eliminate the soft lockup warning on bootup.
> 
> ...
>
> --- a/kernel/watchdog.c
> +++ b/kernel/watchdog.c
> @@ -496,7 +496,9 @@ static void watchdog_enable(unsigned int cpu)
>  		      HRTIMER_MODE_REL_PINNED_HARD);
>  
>  	/* Initialize timestamp */
> -	__touch_watchdog();
> +	if (system_state != SYSTEM_BOOTING)
> +		__touch_watchdog();

A comment which explains the system_state test would be appropriate
here.

>  	/* Enable the perf event */
>  	if (watchdog_enabled & NMI_WATCHDOG_ENABLED)
>  		watchdog_nmi_enable(cpu);