linux-kernel - [PATCH v2 0/4] PCI/AER: Handle Advisory Non-Fatal properly

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240125062802.50819-1-qingshun.wang@linux.intel.com>
Date: Thu, 25 Jan 2024 14:27:58 +0800
From: "Wang, Qingshun" <qingshun.wang@...ux.intel.com>
To: linux-pci@...r.kernel.org,
	linuxppc-dev@...ts.ozlabs.org,
	linux-acpi@...r.kernel.org
Cc: chao.p.peng@...ux.intel.com,
	erwin.tsaur@...el.com,
	feiting.wanyan@...el.com,
	qingshun.wang@...el.com,
	"Rafael J. Wysocki" <rafael@...nel.org>,
	Len Brown <lenb@...nel.org>,
	James Morse <james.morse@....com>,
	Tony Luck <tony.luck@...el.com>,
	Borislav Petkov <bp@...en8.de>,
	Davidlohr Bueso <dave@...olabs.net>,
	Jonathan Cameron <jonathan.cameron@...wei.com>,
	Dave Jiang <dave.jiang@...el.com>,
	Alison Schofield <alison.schofield@...el.com>,
	Vishal Verma <vishal.l.verma@...el.com>,
	Ira Weiny <ira.weiny@...el.com>,
	Dan Williams <dan.j.williams@...el.com>,
	Bjorn Helgaas <bhelgaas@...gle.com>,
	Bjorn Helgaas <helgaas@...nel.org>,
	Mahesh J Salgaonkar <mahesh@...ux.ibm.com>,
	"Oliver O'Halloran" <oohall@...il.com>,
	Miaohe Lin <linmiaohe@...wei.com>,
	Shiju Jose <shiju.jose@...wei.com>,
	Adam Preble <adam.c.preble@...el.com>,
	Li Yang <leoyang.li@....com>,
	Lukas Wunner <lukas@...ner.de>,
	Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@...ux.intel.com>,
	Smita Koralahalli <Smita.KoralahalliChannabasappa@....com>,
	Robert Richter <rrichter@....com>,
	linux-kernel@...r.kernel.org,
	linux-cxl@...r.kernel.org,
	linux-edac@...r.kernel.org,
	"Wang, Qingshun" <qingshun.wang@...ux.intel.com>
Subject: [PATCH v2 0/4] PCI/AER: Handle Advisory Non-Fatal properly

According to PCIe Base Specification Revision 6.1, Sections 6.2.3.4
and 6.2.4.3, certain uncorrectable errors will signal ERR_COR instead
of ERR_NONFATAL, logged as Advisory Non-Fatal Error, and set bits in
both Correctable Error Status register and Uncorrectable Error Status
register. Currently, when handling AER events the kernel will only look
at CE status or UE status, but never both. In the Advisory
Non-Fatal Error case, bits set in the UE status register will not be
reported and cleared until the next Fatal/Non-Fatal error arrives.

For instance, previously, when the kernel receives an ANFE with Poisoned
TLP in OS native AER mode, only the status of CE will be reported and
cleared:

  AER: Corrected error received: 0000:b7:02.0
  PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
    device [8086:0db0] error status/mask=00002000/00000000
     [13] NonFatalErr           

If the kernel receives a Malformed TLP after that, two UE will be
reported, which is unexpected. The Malformed TLP Header was lost since
the previous ANFE gated the TLP header logs:

  PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
    device [8086:0db0] error status/mask=00041000/00180020
     [12] TLP                    (First)
     [18] MalfTLP       

To handle this case properly, add additional fields in aer_err_info to
track both CE and UE status/mask, UE severity, and Device Status.
Use this information to determine the status bits that need to be cleared.

Now, in the previous scenario, both CE status and related UE status will
be reported and cleared after ANFE:

  AER: Corrected error received: 0000:b7:02.0
  PCIe Bus Error: severity=Corrected, type=Transaction Layer, (Receiver ID)
    device [8086:0db0] error status/mask=00002000/00000000
     [13] NonFatalErr           
    Uncorrectable errors that may cause Advisory Non-Fatal:
     [18] TLP

Additionally, add more data to aer_event tracepoint, which would help
to better understand ANFE and other errors for external observation.

Note:
checkpatch.pl will produce following errors and warnings on PATCH 4:

  ERROR: space prohibited after that open parenthesis '('
  #103: FILE: include/ras/ras_event.h:319:
  +		__field(	u16,		link_status	)

  ...similar errors omitted...

  WARNING: quoted string split across lines
  #126: FILE: include/ras/ras_event.h:342:
  +	TP_printk("%s PCIe Bus Error: severity=%s, %s, TLP Header=%s, "
  +		 "Correctable Error Status=0x%08x, "

  ...similar warnings omitted...

For readability reasons, these errors and warnings are not fixed,
following the code style of existing examples in the kernel source tree.

Change log:
v2:
  - Reference to the latest PCIe Specification in both commit messages
    and comments, as suggested by Bjorn Helgaas.
  - Describe the reason for storing additional information in
    aer_err_info in the commit message of PATCH 1, as suggested by Bjorn
    Helgaas.
  - Add more details of behavior changes in the commit message of PATCH
    2, as suggested by Bjorn Helgaas.

v1: https://lore.kernel.org/linux-pci/20240111073227.31488-1-qingshun.wang@linux.intel.com/

Wang, Qingshun (4):
  PCI/AER: Store more information in aer_err_info
  PCI/AER: Handle Advisory Non-Fatal properly
  PCI/AER: Fetch information for FTrace
  RAS: Trace more information in aer_event

 drivers/acpi/apei/ghes.c      |  16 ++-
 drivers/cxl/core/pci.c        |  15 ++-
 drivers/pci/pci.h             |  12 ++-
 drivers/pci/pcie/aer.c        | 191 +++++++++++++++++++++++++++-------
 include/linux/aer.h           |   6 +-
 include/linux/pci.h           |  27 +++++
 include/ras/ras_event.h       |  48 ++++++++-
 include/uapi/linux/pci_regs.h |   1 +
 8 files changed, 269 insertions(+), 47 deletions(-)


base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
-- 
2.42.0