linux-kernel - Re: Hardware Error Kernel Mini-Summit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <m1wrv0zo9q.fsf@fess.ebiederm.org>
Date:	Tue, 18 May 2010 18:14:09 -0700
From:	ebiederm@...ssion.com (Eric W. Biederman)
To:	Andi Kleen <andi@...stfloor.org>
Cc:	Borislav Petkov <bp@...64.org>,
	"Luck\, Tony" <tony.luck@...el.com>,
	Hidetoshi Seto <seto.hidetoshi@...fujitsu.com>,
	Mauro Carvalho Chehab <mchehab@...hat.com>,
	"Young\, Brent" <brent.young@...el.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...hat.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Matt Domsch <Matt_Domsch@...l.com>,
	Doug Thompson <dougthompson@...ssion.com>,
	Joe Perches <joe@...ches.com>, Ingo Molnar <mingo@...e.hu>,
	"bluesmoke-devel\@lists.sourceforge.net" 
	<bluesmoke-devel@...ts.sourceforge.net>,
	Linux Edac Mailing List <linux-edac@...r.kernel.org>
Subject: Re: Hardware Error Kernel Mini-Summit

Andi Kleen <andi@...stfloor.org> writes:

> The original motivation to put them somewhere else
> because I was sick of people reporting them as kernel bugs.

This suggests that to get things reported in dmesg I should
setup a cron job that pulls the latest kernel checks to see
if things are reported into syslog and sends you an email
if things are wrong.

I'm not ready to believe the average person that is running linux
is too stupid to understand the difference between a hardware
error and a software error.

> But there's more to it now:
>
>> If your system isn't broken correctable errors are rare.  People look
>
> Actually the more memory you have the more common they are.
> And the trend is to more and more memory.

The error rate should not be fixed per bit but should be roughly fixed
per DIMM.  If the error rate over time is fixed per bit we are in deep
trouble.

> Really to do anything useful with them you need trends
> and automatic actions (like predictive page offlining)

Not at all, and I don't have a clue where you start thinking
predictive page offlining makes the least bit of sense.  Broken
or even weak bits are rarely the common reason for ECC errors.

> A log isn't really a good format for that

A log is a fine format for realizing you have a problem.  A
log doesn't need to be the only place errors are reported
but a log should be the default place ECC errors are reported.
We do that with hard drive errors and other kinds of hardware
errors and we have done it for years without problems.

My experience is that correctable ECC errors come in two kinds of
frequencies.

- The expected single bit correctable error range.  Which is somewhere
  between once a month and once a year per dimm.

  On the most unreasonable configuration I ever worked with. 4TB of ram
  in 1GB sticks up at Los Alomos, at 7000ft in an environment know
  to trigger errors I saw roughly one correctable ECC error an hour.
  Huge but just barely within the expected range.

  I can live with a log message once a month on a mundane system.

- Errors that occur frequently. That is broken hardware of one time or
  another.  I want to know about that so I can schedule down time to replace
  my memory before I get an uncorrected ECC error.  Errors of this kind
  are likely happening frequently enough as to impact performance.

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/