lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20250728081748.700ad46a@kernel.org>
Date: Mon, 28 Jul 2025 08:17:48 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: Tariq Toukan <ttoukan.linux@...il.com>
Cc: Tariq Toukan <tariqt@...dia.com>, Eric Dumazet <edumazet@...gle.com>,
 Paolo Abeni <pabeni@...hat.com>, Andrew Lunn <andrew+netdev@...n.ch>,
 "David S. Miller" <davem@...emloft.net>, Jiri Pirko <jiri@...nulli.us>,
 Jiri Pirko <jiri@...dia.com>, Saeed Mahameed <saeed@...nel.org>, Gal
 Pressman <gal@...dia.com>, Leon Romanovsky <leon@...nel.org>, Shahar
 Shitrit <shshitrit@...dia.com>, Donald Hunter <donald.hunter@...il.com>,
 Jonathan Corbet <corbet@....net>, Brett Creeley <brett.creeley@....com>,
 Michael Chan <michael.chan@...adcom.com>, Pavan Chebbi
 <pavan.chebbi@...adcom.com>, Cai Huoqing <cai.huoqing@...ux.dev>, Tony
 Nguyen <anthony.l.nguyen@...el.com>, Przemek Kitszel
 <przemyslaw.kitszel@...el.com>, Sunil Goutham <sgoutham@...vell.com>, Linu
 Cherian <lcherian@...vell.com>, Geetha sowjanya <gakula@...vell.com>, Jerin
 Jacob <jerinj@...vell.com>, hariprasad <hkelam@...vell.com>, Subbaraya
 Sundeep <sbhatta@...vell.com>, Saeed Mahameed <saeedm@...dia.com>, Mark
 Bloch <mbloch@...dia.com>, Ido Schimmel <idosch@...dia.com>, Petr Machata
 <petrm@...dia.com>, Manish Chopra <manishc@...vell.com>,
 netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
 linux-doc@...r.kernel.org, intel-wired-lan@...ts.osuosl.org,
 linux-rdma@...r.kernel.org
Subject: Re: [PATCH net-next 0/5] Expose grace period delay for devlink
 health reporter

On Sun, 27 Jul 2025 14:00:11 +0300 Tariq Toukan wrote:
> I get your suggestion. I agree that it's also pretty simple to 
> implement, and that it tolerates bursts.
> 
> However, I think it softens the grace period role too much. It has an 
> important disadvantage, as it tolerates non-bursts as well. It lacks the 
> "burstness" distinguishability.
> 
> IMO current grace_period has multiple goals, among them:
> 
> a. let the auto-recovery mechanism handle errors as long as they are 
> followed by some long-enough "healthy" intervals.
> 
> b. break infinite loop of auto-recoveries, when the "healthy" interval 
> is not long enough. Raise a flag to mark the need for admin intervention.
> 
> In your proposal, the above doesn't hold.
> It won't prevent the infinite auto-recovery loop for a buggy system that 
> has a constant rate of up to X failures in N msecs.
> 
> One can argue that this can be addressed by increasing the grace_period. 
> i.e. a current system with grace_period=N is intuitively moved to 
> burst_size=X and grace_period=X*N.
> 
> But increasing the grace_period by such a large factor has 
> over-enforcement and hurts legitimate auto-recoveries.
> 
> Again, the main point is, it lacks the ability to properly distinguish 
> between 1. a "burst" followed by a healthy interval, and 2. a buggy 
> system with a rate of repeated errors.

I suspect this is catching some very mlx5-specific recovery loop,
so I defer to your judgment on what's better.

As a user I do not know how to configure this health recovery stuff.
My intuition would be that we just needs to lower the recovery rate
to prevent filling up logs etc. and the action of taking the machine
out is really the responsibility of some fleet health monitoring daemon.
I can't think of any other error reporting facility in the kernel where
we'd shut down the recovery completely if the rate is high..

> > Now we add something called "grace period delay" in
> > some places in the code referred to as "reporter_delay"..
> > 
> > It may be more palatable if we named the first period "error burst
> > period" and, well, the later I suppose it's too late to rename..  
> It can be named after what it achieves (allows handling of more errors) 
> or what it is (a shift of the grace_period). I'm fine with both, don't 
> have strong preference.

Let's rename to "error burst period", reporter_in_error_burst etc.

> I'd call it grace_period in case we didn't have one already :)

Exactly :)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ