Message-ID: <m1fwe9bpk2.fsf@fess.ebiederm.org>
Date:	Fri, 17 Feb 2012 04:41:01 -0800
From:	ebiederm@...ssion.com (Eric W. Biederman)
To:	Don Zickus <dzickus@...hat.com>
Cc:	Yinghai Lu <yinghai@...nel.org>, linux-kernel@...r.kernel.org,
	mingo@...hat.com, hpa@...or.com, torvalds@...ux-foundation.org,
	kexec@...ts.infradead.org, vgoyal@...hat.com,
	akpm@...ux-foundation.org, tglx@...utronix.de, mingo@...e.hu,
	linux-tip-commits@...r.kernel.org
Subject: Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

ebiederm@...ssion.com (Eric W. Biederman) writes:

> Don Zickus <dzickus@...hat.com> writes:
>
>> On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
>>> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus@...hat.com> wrote:
>>> 
>>> > So I think I figured it out.  I went through and commented out code in
>>> > disable_local_APIC until I narrowed it down to the piece of code that
>>> > needs to be disabled for it to work.
>>> >
>>> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
>>> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
>>> > is generated by the local apic during decompression (just bad timing) and
>>> > *splat*.
>>> >
>>> > Yinghai, you can probably prove this by
>>> >
>>> > echo 0 > /proc/sys/kernel/nmi_watchdog
>>> >
>>> > then do your kdump crash test.
>>> 
>>> yes.  that will make kdump crash working.
>>
>> Cool.  Thanks.
>>
>> Eric,
>>
>> Just let me know how you want to handle disabling NMIs in the kexec in
>> panic shutdown case.
>
> Interesting.  Apparently we have been avoiding this problem by accident.
>
> Thanks for hunting this down.
>
> The options I can see are:
> - Ensure we can handle and ignore exceptions like this.
> - Always shutoff the lapic and ioapic entries that can generate this.
>
> The good news is that both solutions should be lock free.
>
> The current kernel boot code relies on the assumption that all
> interrupts can be disabled.  In this case with nmi's that is clearly not
> the case.
>
> The most robust solution and what we want to do long term is to
> install an idt that will simply ignore all interrupts until the
> idt is replaced.  Since really all we need to deal with is the NMI
> vector, which is vector #2, we can have a very small interrupt
> descriptor table.
>
> Unfortunately we go through some cpu mode switches in /sbin/kexec,
> allowing us to enter the kernels 32bit entry point before we
> run the decompresser, so at first glance both /sbin/kexec and the
> kernel need to be fixed in a coordinated fashion.
>
> There are two was I can see of removing the need for an exactly
> coordinated release.
> - Document that an old /sbin/kexec userspace requires you not to
>   use the nmi watchdog with modern kernels.
> - For a short while simply retain code that stomps the nmi watchdog.
>   (But still leaves us open to other kinds of nmi's).
>
> Grr.  Looking a little more closely, all throughout the linux kernel's
> boot there is the assumption that any interrupt during boot is a failure
> of some kind, and except for an errant nmi watchdog that is a true
> assumption.
>
> Don I guess I really have to recommend disabling the nmi watchdog in the
> kexec on panic path if we can do so at all reasonably. 
>
> I like the idea of ignoring nmis during boot but that seems to be a
> slightly larger project and with little practical improvement in kexec
> on panic quality.  Other than getting what should be one or two
> i/o writes out of the kexec on panic path.

Hmm.

Thinking about it a little more.  The kernel's boot path is inconsistent
with the rest of the kernel's nmi handling.  For anything exception
except an nmi stopping and giving up is fine.  For an nmi it is very
rare for an NMI to signal a truly nasty failure (usually it just means
someone saw a bitflip somewhere), and we can almost always continue
without problem.

I think in practice we really should make our boot path consistent with
the rest of the kernel and ignore/log/report nmis but not fail on them.
Triple faulting (trigger a cpu reset) as we do today just seems like a
recipe for deep and confusing mystery, and not being helpful to the
user.

My preferred fix would be to fix the boot path and /sbin/kexec to ignore
and report nmis as we boot, as that is really what we want long term and
it gives us the most robust solution.

The fix with a guarantee of no more scope creep is to just disable the
nmi watchdog on the kexec on panic path.

Don if you have time please figure out is needed to ignore nmi's and
possible record and/or report them while we boot, otherwise please cook
up a patch that just disables the nmi watchdog wherever we are sending
it from (the local apic or the ioapic).

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists