linux-kernel - Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 11 Oct 2018 17:59:41 +0800
From:   Feng Tang <feng.tang@...el.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>,
        H Peter Anvin <hpa@...ux.intel.com>,
        Borislav Petkov <bp@...en8.de>,
        Andrew Morton <akpm@...ux-foundation.org>,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

Hi Peter,

On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> > 
> > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [   35.749975] ------------[ cut here ]------------
> > 
> > These messages may overflow the sceen (framebuffer) and make debugging
> > much difficulter.
> 
> *blink* you actually using the framebuffer for debugging ?! Why the heck
> are you doing that?

I didn't make it clear.

I just wanted to show some use case, that when I work under command line
mode, if the kernel panic (without reboot configured), it will pop out
the panic message, and these noisy message may keep poping up and I
hardly see the actual panic message on the screen.

> 
> > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > index c93fcfd..b703862 100644
> > --- a/arch/x86/kernel/process.c
> > +++ b/arch/x86/kernel/process.c
> > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> >  	 * Remove this CPU:
> >  	 */
> >  	set_cpu_online(smp_processor_id(), false);
> > +	set_cpu_active(smp_processor_id(), false);
> >  	disable_local_APIC();
> >  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> >  
> 
> WTH is stop_this_cpu() and how do we even get here with active still
> set?

Current panic() will call
	smp_send_stop
		native_stop_other_cpus
			apic->send_IPI_allbutself(REBOOT_VECTOR)
				smp_reboot_interrupt
					stop_this_cpu

> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 7fc4a37..cf41b7b 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> >  {
> >  	int ilb = cpumask_first(nohz.idle_cpus_mask);
> >  
> > -	if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > +	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> >  		return ilb;
> >  
> >  	return nr_cpu_ids;
> 
> 
> Similar, this is the result of taking the CPU away without going through
> the normal path. You're doing something dodgy.

You are right, it's hacky. As I said in the commit log, suggestions are
appreciated.

Current emergercy handling of panic didn't notify the idle subsystem of
other cpus being stopped.

I have tried several other ways:
a) add a panic notifier to cancel the schdule timer when panic happens
b) don't re-enable the local interrupt inside panic() after normal panic
handling.

Thanks,
Feng