linux-kernel - Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20120221135934.GF26998@redhat.com>
Date:	Tue, 21 Feb 2012 08:59:34 -0500
From:	Don Zickus <dzickus@...hat.com>
To:	"Eric W. Biederman" <ebiederm@...ssion.com>
Cc:	Yinghai Lu <yinghai@...nel.org>, linux-kernel@...r.kernel.org,
	mingo@...hat.com, hpa@...or.com, torvalds@...ux-foundation.org,
	kexec@...ts.infradead.org, vgoyal@...hat.com,
	akpm@...ux-foundation.org, tglx@...utronix.de, mingo@...e.hu,
	linux-tip-commits@...r.kernel.org
Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in
 crash path

On Tue, Feb 21, 2012 at 12:01:07AM -0800, Eric W. Biederman wrote:
> Don Zickus <dzickus@...hat.com> writes:
> 
> 2> On Fri, Feb 17, 2012 at 07:21:52PM -0800, Eric W. Biederman wrote:
> >> Don Zickus <dzickus@...hat.com> writes:
> >> 
> >> > On Fri, Feb 17, 2012 at 04:41:01AM -0800, Eric W. Biederman wrote:
> >> >> 
> >> >> The fix with a guarantee of no more scope creep is to just disable the
> >> >> nmi watchdog on the kexec on panic path.
> >> >> 
> >> >> Don if you have time please figure out is needed to ignore nmi's and
> >> >> possible record and/or report them while we boot, otherwise please cook
> >> >> up a patch that just disables the nmi watchdog wherever we are sending
> >> >> it from (the local apic or the ioapic).
> >> >
> >> > Can I keep things even simpler?  The original problem was the deadlock
> >> > with the ioapic lock.  We fixed that by removing the call to
> >> > disable_IO_APIC().  Can we just leave the disable_local_APIC calls in
> >> > there for now?  Is there any real harm?
> >> 
> >> > All this rewrite is going to take time which will delay fixing a current
> >> > problem with kexec on panic, the ioapic deadlock.
> >> 
> >> Hmm.
> >> 
> >> My apologies I just realized that we can not disable the nmi watchdog
> >> safely in all cases.  To avoid the deadlock we fundamentally can not
> >> write to the io_apic, because the locks are the io_apic write path.
> >> The nmi watchdog can be sourced from either the local apics or the
> >> io_apics.  To disable the nmi_watchdog we need at least potentially
> >> to write io_apic.
> >
> > I am curious where you see the nmi watchdog being sourced from the ioapic?
> > I thought I removed that code 3 or 4 releases ago.
> 
> In my memory, and in references to the code in comments in various apic
> related code.  I couldn't figure out what the current code was doing and
> assumed the implementation was equivalent.  It does look like you
> removed the code that used the io_apic.  I still haven't figured
> out just how the new implementation works yet.

It is just a client of perf now.  Perf controls the local apic accesses.
A lot simpler now.

> 
> So maybe in the short term we can safely just stomp the timer that
> triggers the nmi watchdog in the local apic.  Over the long term that
> feels like it is just asking for trouble.

I can understand that.

> 
> I wonder if the reason that we have an hpet stomp in that code is
> for a similar reason.  Did we ever source nmi's from the hpet timer?
> 
> >> So it appears to me that the only reasonable and robust thing we can
> >> do is to ignore nmis in the kexec on panic path.
> >> 
> >> So it looks to me that the only path forward at this point is to fix
> >> the other bug where an unexpected nmi will kill the kexec on panic boot.
> >> 
> >> I just took a look at the code in /sbin/kexec and that code does not in
> >> fact change the idt except when we switch to 16bit mode, which we
> >> definitely do not do in the kexec on panic case.  So it appears that we
> >> don't need to coordinate an /sbin/kexec release with a kernel release to
> >> ignore nmis.
> >> 
> >> In fact it looks like we only need to fix the interrupt descriptors
> >> loaded in machine_kexec_64.c and head64.c to ignore nmis.
> >> 
> >> At which point we will have fixed two bugs and have a much more reliable
> >> kexec on panic implementation.
> >
> > Ok.  I'll talk with Vivek about how the can be implemented.
> 
> Thanks.  It really doesn't look very hard.  Just a tiny idt with
> an nmi entry that says iret.

It probably is, except I never hacked on idt code before and my assembly
isn't that good.  I have been trying to find examples to copy from to give
it a try.  So far I was using early_idt_handlers with early_printk to see
if I could capture some printk messages while jumping from the first
kernel to the second kernel (when the other early_idt_handlers would kick
in for the second kernel).

Tips?  Better examples?

Cheers,
Don
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/