linux-kernel - RE: [PATCH 3/3] mce: acpi/apei: trace: Enable ghes memory error trace event

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <3908561D78D1C84285E8C5FCA982C28F31CBAAFA@ORSMSX106.amr.corp.intel.com>
Date:	Wed, 14 Aug 2013 18:38:09 +0000
From:	"Luck, Tony" <tony.luck@...el.com>
To:	Borislav Petkov <bp@...en8.de>
CC:	"Naveen N. Rao" <naveen.n.rao@...ux.vnet.ibm.com>,
	Mauro Carvalho Chehab <m.chehab@...sung.com>,
	"bhelgaas@...gle.com" <bhelgaas@...gle.com>,
	"rostedt@...dmis.org" <rostedt@...dmis.org>,
	"rjw@...k.pl" <rjw@...k.pl>,
	"lance.ortiz@...com" <lance.ortiz@...com>,
	"linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
	"linux-acpi@...r.kernel.org" <linux-acpi@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: [PATCH 3/3] mce: acpi/apei: trace: Enable ghes memory error
 trace event

> Didn't we say at some point, "log only the panic messsage which kills
> the machine"?

We've wandered around different strategies here. We definitely want
the panic log.  Some people want all other "kernel exit" logs (shutdown,
reboot, kexec).  When there is enough space in the pstore backend we
might also want the "oops" that preceeded the panic.  (Of course when
the oops happens we don't know the future, so have to save it just in
case ... then if more "oops" happen we have to decide whether to keep
the old oops log, or save the newer one).

> However, we probably could use more the messages before that
> catastrophic event because they could give us hints about what lead to
> the panic but in that case maybe a limited pstore is the wrong logging
> medium.
Yes - longer logs are better.  Sad that the pstore backend devices are
measured in kilobytes :-)

> Actually, I can imagine the full serial/network logs of "special"
> tracepoints + dmesg to be the optimal thing.
If you guess the right "special" tracepoints to log - then yes.

> Actually, does the pstore act like a circular buffer? Because if it
> contains the last N relevant messages (for an arbitrary definition of
> relevant) before the system dies, then that could more helpful than only
> the error messages.
No - write speed for the persistent storage backing pstore (flash) means we
don't log as we go. We wait for a panic and then our registered function
gets called so we can snapshot what is in the console log at that point.
We also don't want to wear out the flash which may be soldered to the
motherboard.


> Ok, let's sort:

> * First we have the not-so-critical hw error messages. We want to carry
> those out-of-band, i.e. not in dmesg so that people don't have to parse
> and collect dmesg but have a specialized solution which gives them
> structured logs and tools can analyze, collect and ... those errors.

Agreed - we shouldn't clutter logs with details of corrected errors.
At most we should have a rate-limited log showing the count of corrected errors
so that someone who just watches dmesg knows they should go dig deeper
if they see some big number of corrected errors.

> * When a critical error happens, the above usage is not necessarily
> advantageous anymore in the sense that, in order to debug what caused
> the machine to crash, we don't simply necessarily want only the crash
> message but also the whole system activity that lead to it.

Yes.  There are people looking at various "flight recorder" modes for tracing
that keep logs of normal events in a circular buffer in RAM ... if these exist
they should be saved at crash time (and they are in the kexec/kdump path,
but I don’t know if anyone does anything in the non-kdump case).

> In which case, we probably actually want to turn off/ignore the error
> logging tracepoints and write *only* to dmesg which goes out over serial
> and to pstore. Right?
Tracepoints for errors that are going to lead to system crash would only be
useful together with a flight recorder to make sure they get saved. I think
tracepoints for corrected errors are better than dmesg logs.

> Because in such cases I want to have *all* *relevant* messages that lead
> to the explosion + the explosion message itself.

In a perfect world yes - I don't know that we can achieve perfection - but we
can iterate through good, better, even better.  The really hard part of this is
figuring out what is *relevant* to save before a particular crash happens.

-Tony