linux-kernel - Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20181022095515.wxq6dhvix4ujkskv@shbuild888>
Date:   Mon, 22 Oct 2018 17:55:15 +0800
From:   Feng Tang <feng.tang@...el.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>,
        H Peter Anvin <hpa@...ux.intel.com>,
        Borislav Petkov <bp@...en8.de>,
        Andrew Morton <akpm@...ux-foundation.org>,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

Hi maintainers,

On Thu, Oct 11, 2018 at 05:59:41PM +0800, Feng Tang wrote:
> Hi Peter,
> 
> On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> > On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > > Sometimes when debugging kernel panic, we saw many extra noisy error
> > > messages after the expected end:
> > > 
> > > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > > [   35.749975] ------------[ cut here ]------------
> > > 
> > > These messages may overflow the sceen (framebuffer) and make debugging
> > > much difficulter.
> > 
> > *blink* you actually using the framebuffer for debugging ?! Why the heck
> > are you doing that?
> 
> I didn't make it clear.
> 
> I just wanted to show some use case, that when I work under command line
> mode, if the kernel panic (without reboot configured), it will pop out
> the panic message, and these noisy message may keep poping up and I
> hardly see the actual panic message on the screen.
> 
> > 
> > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > > index c93fcfd..b703862 100644
> > > --- a/arch/x86/kernel/process.c
> > > +++ b/arch/x86/kernel/process.c
> > > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> > >  	 * Remove this CPU:
> > >  	 */
> > >  	set_cpu_online(smp_processor_id(), false);
> > > +	set_cpu_active(smp_processor_id(), false);
> > >  	disable_local_APIC();
> > >  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> > >  
> > 
> > WTH is stop_this_cpu() and how do we even get here with active still
> > set?
> 
> Current panic() will call
> 	smp_send_stop
> 		native_stop_other_cpus
> 			apic->send_IPI_allbutself(REBOOT_VECTOR)
> 				smp_reboot_interrupt
> 					stop_this_cpu
> 
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 7fc4a37..cf41b7b 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> > >  {
> > >  	int ilb = cpumask_first(nohz.idle_cpus_mask);
> > >  
> > > -	if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > > +	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> > >  		return ilb;
> > >  
> > >  	return nr_cpu_ids;
> > 
> > 
> > Similar, this is the result of taking the CPU away without going through
> > the normal path. You're doing something dodgy.
> 
> You are right, it's hacky. As I said in the commit log, suggestions are
> appreciated.
> 
> Current emergercy handling of panic didn't notify the idle subsystem of
> other cpus being stopped.
> 
> I have tried several other ways:
> a) add a panic notifier to cancel the schdule timer when panic happens
> b) don't re-enable the local interrupt inside panic() after normal panic
> handling.

Could you help to review the patch? thanks,

- Feng