linux-kernel - [PATCH 0/2] panic: taint flag for recoverable hardware errors

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20250704-taint_recovered-v1-0-7a817f2d228e@debian.org>
Date: Fri, 04 Jul 2025 03:55:18 -0700
From: Breno Leitao <leitao@...ian.org>
To: Len Brown <lenb@...nel.org>, James Morse <james.morse@....com>, 
 Jonathan Corbet <corbet@....net>, tony.luck@...el.com, rafael@...nel.org
Cc: Alexei Starovoitov <ast@...nel.org>, kbusch@...nel.org, rmikey@...a.com, 
 kuba@...nel.org, ast@...nel.org, linux-edac@...r.kernel.org, 
 mchehab@...nel.org, bp@...en8.de, linux-acpi@...r.kernel.org, 
 linux-kernel@...r.kernel.org, linux-doc@...r.kernel.org, 
 kernel-team@...a.com, Breno Leitao <leitao@...ian.org>
Subject: [PATCH 0/2] panic: taint flag for recoverable hardware errors

Overview
========

This patchset introduces a new kernel taint flag to track systems that
have experienced recoverable hardware errors during runtime. The
motivation comes from the operational challenges of managing large
server fleets where hardware events are common, and having them tainting
the kernel helps operators to correlate problems more easily.

This complement the new MACHINE_CHECK taint that got added for fatal
errors. [1]

Problem Statement
=================

In large-scale deployments with thousands of servers, hardware errors
are inevitable. While modern systems can recover from many hardware
failures (corrected ECC errors, recoverable CPU errors, etc.), these
events causes the kernel to behave in very different ways, which can
cause  bugs due to the path that is rarely exercised.

I experienced this pain very recently, where several machines were
crashing due to a recoverable PCI offline port. The hardware was
behaving correctly, but, during the recoverable process, the kernel goes
through some code path that is rarely tested.

In my case, the kernel recoverable process caused some issues that were
hard to find the root cause. For instance, recoverable PCI events
cause the device to suddently go offline, and later PCI re-enumeration,
which would reinitalize the driver.

The event above caused some real crashes in production, in very
different ways. From those that I investigated, I found:

	1) If the disk was going away, it was causing a file systems
	   issue that got already fixed in 6.14 and 6.15

	2) If the network was going away, it was causing some iommu
	   issues discussed and fixed in [2].

	3) Possible other issues, that were not easy to correlate, such
	   as stalls, hungup tasks, memory leaks, warnings, etc.

	  a) These are hidden today, and I would like to expose them
	     with this patch.

In summary, when investigating system issues, there's no trivial way to
determine if a machine has previously experienced hardware problems that
might be contributing to current instability, other than going host by
host and scanning kernel logs.

Proposed Solution
=================

Add a new taint flag to the kernel (HW_ERROR_RECOVERED - for the lack of
a better name) that gets set whenever the kernel detects and recovers
from hardware errors.

The taint provides additional context during crash investigation *without*
implying that crashes are necessarily caused by hardware failures
(similar to how PROPRIETARY_MODULE taint works). It is just an extra
information that will provide more context about that machine.

This patchset focuses on ACPI/GHES, which handles most recoverable
hardware errors I have experience with, but can be extended to other
subsystems like EDAC HW_EVENT_ERR_CORRECTED in the future.

--

I would like to *thanks* Tony for the early discussions and
encouragement.

Link: https://lore.kernel.org/all/20250702-add_tain-v1-1-9187b10914b9@debian.org/ [1]
Link: https://lore.kernel.org/all/20250409-page-pool-track-dma-v9-0-6a9ef2e0cba8@redhat.com/ [2]

---
Breno Leitao (2):
      panic: add taint flag for recoverable hardware errors
      acpi/ghes: taint kernel on recovered hardware errors

 Documentation/admin-guide/tainted-kernels.rst | 7 ++++++-
 drivers/acpi/apei/ghes.c                      | 7 +++++--
 include/linux/panic.h                         | 3 ++-
 kernel/panic.c                                | 1 +
 tools/debugging/kernel-chktaint               | 8 ++++++++
 5 files changed, 22 insertions(+), 4 deletions(-)
---
base-commit: dc3cd0dfd91cad0611f0f0eace339a401da5d5ee
change-id: 20250703-taint_recovered-1d2e890a684b

Best regards,
--  
Breno Leitao <leitao@...ian.org>