Message-ID: <20111011184434.GB32316@in.ibm.com>
Date:	Wed, 12 Oct 2011 00:14:34 +0530
From:	"K.Prasad" <prasad@...ux.vnet.ibm.com>
To:	Borislav Petkov <bp@...en8.de>, Vivek Goyal <vgoyal@...hat.com>,
	linux-kernel@...r.kernel.org, crash-utility@...hat.com,
	kexec@...ts.infradead.org, Andi Kleen <andi@...stfloor.org>,
	"Luck, Tony" <tony.luck@...el.com>,
	"Eric W. Biederman" <ebiederm@...ssion.com>, anderson@...hat.com,
	tachibana@....nes.nec.co.jp, oomichi@....nes.nec.co.jp,
	Valdis.Kletnieks@...edu, Nick Bowler <nbowler@...iptictech.com>
Subject: Re: [Patch 1/4][kernel][slimdump] Add new elf-note of type
 NT_NOCOREDUMP to capture slimdump

On Mon, Oct 10, 2011 at 09:07:25AM +0200, Borislav Petkov wrote:
> On Fri, Oct 07, 2011 at 09:42:19PM +0530, K.Prasad wrote:
> > The problem, as pointed out by Borislav Petkov in a different mail, is that
> > we might end up capturing a vmcore containing corrupted data when the
> > same is not required for analysing the cause of the crash.
> > 
> > Of course, all this is assuming that reading the faulty memory with MCE
> > disabled is harmless. However, the effect of a read operation in this
> > case is undefined.
> 
> Frankly, I don't think that it is undefined - you basically should be
> able to read DRAM albeit with the corrupted data in it. However, you
> probably best disable the whole DRAM error detection first by clearing
> a couple of bits in MC4_CTL_MASK (at least on AMD that should work, I
> dunno how Intel does that).
> 

The MC4_CTL_MASK doesn't appear to be defined in the kernel. Looking at
http://support.amd.com/us/Processor_TechDocs/26094.PDF, Page 196, it
states that "This register is typically programmed by BIOS and not by
the Kernel software".

So, in any case we may not be able to disable machine-check exceptions
(MCEs) only within the context of kexec'ed kernel. Let me know if I've
missed something here.

> But, regardless, according to Vivek, the "makedumpfile" tool should be
> able to jump over poisoned pages and you don't need all the hoopla above
> at all, right?
>

In short, the answer is yes. We could add a new string, say
"CRASH_REASON=PANIC_MCE" to VMCOREINFO elf-note which can be parsed by
'makedumpfile' and get away without adding the new NT_NOCOREDUMP
elf-note. Parsing through the log_buf to lookout for panic string from
inside 'makedumpfile' appears to be a clumsy solution though.

The suggestion to make NT_NOCOREDUMP to contain more fine-granular
information can be met by using meaningful strings for VMCOREINFO.

---

In this context, I wish to quickly recollect the issues we've discussed
thus far, their proposed solutions and re-evaluate the need for new elf-note.

i) Scenario1: System crashes because of a fatal MCE

Proposed Solution: Add a new string in the VMCOREINFO elf-note from
within the MCE panic path to indicate cause of crash. 'makedumpfile'
recognises this string to collect a slimdump instead of the normal dump.

ii) Scenario2: System with PG_hwpoison (or landmine!) pages crashes because
of a software bug. In this case, kexec kernel would normally reboot because
of reading the PG_poison page. I'll soon get a new version of the patchset
implementing this.

Solution: Maintain a linked list of PFNs when the corresponding 'struct page'
has been marked PG_hwpoison. We could export/put this list to use in
quite a few ways.

- Make it a policy in the kernel to not operate upon a 'read' request
  for such pages. Return '0' from copy_oldmem_page() function if the PFN
  is part of the PG_hwpoison list. I don't see a reason why anybody
  would be interested in reading the contents of a corrupt page, so
  making it standard kernel behaviour should be acceptable (or so I
  hope :-)).

  The list of PFNs must be exported (How? more on that below) to
  user-space, so that downstream tools such as 'crash' recognise that
  the vmcore (corresponding to PG_hwpoison memory regions) contains
  'distorted' data.

- Export the PG_hwpoison PFN list through a new elf-note. Given that
  the PFN list can be long and of indeterminate size (at compile time),
  I'm not sure if individually adding each PFN to the VMCOREINFO note
  would be a good idea and hence the new elf-note.

  Then teach 'makedumpfile' to recognise these PFNs (by exporting their
  VADDR or somesuch mechanism) and avoid reading those pages from
  /proc/vmcore. Also collect these PFNs and pass it down to 'crash' to
  help it identify the 'distorted' memory locations.

The system in kexec-ed kernel could still crash because of fatal MCEs in
its own memory region or new uncorrected memory errors in the old
kernel's memory (error happened after the crash) and can be potentially
'read' during memory copy operation. However the probability of these
occurrences is assumed to be small given the short lifetime of the
kexec-ed kernel.

While we don't actually need a new elf-note for i), I suspect
it might not be the case for resolving ii).

Kindly let me know your thoughts on this.

Thanks,
K.Prasad

P.S.: A quick definition of terms used above
-------------------------------------------
Fatal or unrecoverable MCE - A Machine Check Exception (MCE) that causes
the system to panic. The exception might be triggered due to a faulty
piece of memory in DIMM or cache. It is triggered due to 'consumption'
(read/write) of a memory location with uncorrected memory error.

PG_hwpoison - This is a page flag (marked in 'struct page') when an
uncorrected memory error is detected (through means such as memory
scrubbing) but is not 'consumed' yet. The page is flagged to prevent it
from re-entering the memory stream. Causes the system to crash when
the page with this flag is consumed.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists