linux-kernel - Re: [PATCH] ghes: Track number of recovered hardware errors

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <b4c39a87-c5a4-4525-b598-61fc28a8dc36@linux.alibaba.com>
Date: Wed, 16 Jul 2025 11:04:28 +0800
From: Shuai Xue <xueshuai@...ux.alibaba.com>
To: Breno Leitao <leitao@...ian.org>
Cc: Borislav Petkov <bp@...en8.de>, Alexander Graf <graf@...zon.com>,
 Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>,
 Peter Gonda <pgonda@...gle.com>, "Luck, Tony" <tony.luck@...el.com>,
 "Rafael J. Wysocki" <rafael@...nel.org>, Len Brown <lenb@...nel.org>,
 James Morse <james.morse@....com>, "Moore, Robert" <robert.moore@...el.com>,
 "linux-acpi@...r.kernel.org" <linux-acpi@...r.kernel.org>,
 "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
 "acpica-devel@...ts.linux.dev" <acpica-devel@...ts.linux.dev>,
 "kernel-team@...a.com" <kernel-team@...a.com>
Subject: Re: [PATCH] ghes: Track number of recovered hardware errors

在 2025/7/16 01:25, Breno Leitao 写道:
> Hello Shuai,
> 
> On Tue, Jul 15, 2025 at 09:46:03PM +0800, Shuai Xue wrote:
>>> It would be really good to sync with other cloud providers here so that we can
>>> do this one solution which fits all. Lemme CC some other folks I know who do
>>> cloud gunk and leave the whole mail for their pleasure.
>>>
>>> Newly CCed folks, you know how to find the whole discussion. :-)
>>>
>>> Thx.
>>
>>
>> For the purpose of counting, how about using the cmdline of rasdaemon?
> 
> How do you manage it at a large fleet of hosts? Do you have rasdaemon
> logging always and how do you correlate with kernel crashes? At Meta, we
> have an a "clues" tag for each crash, and one of the tags is Machine
> Check Exception (MCE), which is parsed from dmesg right now (with the
> regexp I shared earlier).

We deploy rasdaemon on each individual node, and then collect the
rasdaemon logs centrally. At the same time, we collect out-of-band
error logs. We aggregate and count the types and occurrences of errors,
and finally use empirical thresholds for operational alerts. The crash
analysis service consumes these alert messages.

> 
> My plan with this patch is to have a counter for hardware errors that
> would be exposed to the crashdump. So, post-morten analyzes tooling can
> easily query if there are hardware errors and query RAS information in
> the right databases, in case it seems a smoking gun.

I see your point. But does using a single ghes_recovered_errors counter
to track all corrected and non-fatal errors for CPU, memory, and PCIe
really help?

> 
> Do you have any experience with this type of automatic correlation?

Please see my reply above.

> 
> Thanks for your insights,
> --breno

Thanks.
Shuai