linux-kernel - Re: [PATCH v6 00/16] Rate limit AER logs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250520090511.GC261485@rocinante>
Date: Tue, 20 May 2025 18:05:11 +0900
From: Krzysztof Wilczyński <kw@...ux.com>
To: Bjorn Helgaas <helgaas@...nel.org>
Cc: linux-pci@...r.kernel.org, Jon Pan-Doh <pandoh@...gle.com>,
	Karolina Stolarek <karolina.stolarek@...cle.com>,
	Martin Petersen <martin.petersen@...cle.com>,
	Ben Fuller <ben.fuller@...cle.com>,
	Drew Walton <drewwalton@...rosoft.com>,
	Anil Agrawal <anilagrawal@...a.com>,
	Tony Luck <tony.luck@...el.com>,
	Ilpo Järvinen <ilpo.jarvinen@...ux.intel.com>,
	Sathyanarayanan Kuppuswamy <sathyanarayanan.kuppuswamy@...ux.intel.com>,
	Lukas Wunner <lukas@...ner.de>,
	Jonathan Cameron <Jonathan.Cameron@...wei.com>,
	Sargun Dhillon <sargun@...a.com>,
	"Paul E . McKenney" <paulmck@...nel.org>,
	Mahesh J Salgaonkar <mahesh@...ux.ibm.com>,
	Oliver O'Halloran <oohall@...il.com>,
	Kai-Heng Feng <kaihengf@...dia.com>,
	Keith Busch <kbusch@...nel.org>, Robert Richter <rrichter@....com>,
	Terry Bowman <terry.bowman@....com>,
	Shiju Jose <shiju.jose@...wei.com>,
	Dave Jiang <dave.jiang@...el.com>, linux-kernel@...r.kernel.org,
	linuxppc-dev@...ts.ozlabs.org, Bjorn Helgaas <bhelgaas@...gle.com>
Subject: Re: [PATCH v6 00/16] Rate limit AER logs

Hello,

> This work is mostly due to Jon Pan-Doh and Karolina Stolarek.  I rebased
> this to v6.15-rc1, factored out some of the trace and statistics updates,
> and added some minor cleanups.
> 
> Proposal
> ========
> 
> When using native AER, spammy devices can flood kernel logs with AER errors
> and slow/stall execution. Add per-device per-error-severity ratelimits for
> more robust error logging. Allow userspace to configure ratelimits via
> sysfs knobs.
> 
> Motivation
> ==========
> 
> Inconsistent PCIe error handling, exacerbated at datacenter scale (myriad
> of devices), affects repairabilitiy flows for fleet operators.
> 
> Exposing PCIe errors/debug info in-band for a userspace daemon (e.g.
> rasdaemon) to collect/pass on to repairability services will allow for more
> predictable repair flows and decrease machine downtime.
> 
> Background
> ==========
> 
> AER error spam has been observed many times, both publicly (e.g. [1], [2],
> [3]) and privately. While it usually occurs with correctable errors, it can
> happen with uncorrectable errors (e.g. during new HW bringup).
> 
> There have been previous attempts to add ratelimits to AER logs ([4], [5]).
> The most recent attempt[5] has many similarities with the proposed
> approach.

I have been testing this series locally with and without faults triggered
using the AER error injection facility.  No issues thus far.

And, as such...

Tested-by: Krzysztof Wilczyński <kwilczynski@...nel.org>

Thank you!

	Krzysztof