netdev - RE: Query about the impact of using CONFIG_PREEMPT_RT on locking mechanisms within networking drivers

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID:
 <TYCPR01MB120933E9C8A96EE9CF617CF1BC2E8A@TYCPR01MB12093.jpnprd01.prod.outlook.com>
Date: Wed, 15 Oct 2025 15:48:46 +0000
From: Fabrizio Castro <fabrizio.castro.jz@...esas.com>
To: Steven Rostedt <rostedt@...dmis.org>
CC: "davem@...emloft.net" <davem@...emloft.net>, "edumazet@...gle.com"
	<edumazet@...gle.com>, "kuba@...nel.org" <kuba@...nel.org>,
	"pabeni@...hat.com" <pabeni@...hat.com>, "andrew@...n.ch" <andrew@...n.ch>,
	"bigeasy@...utronix.de" <bigeasy@...utronix.de>, "clrkwllms@...nel.org"
	<clrkwllms@...nel.org>, "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"linux-rt-devel@...ts.linux.dev" <linux-rt-devel@...ts.linux.dev>, Prabhakar
 Mahadev Lad <prabhakar.mahadev-lad.rj@...renesas.com>
Subject: RE: Query about the impact of using CONFIG_PREEMPT_RT on locking
 mechanisms within networking drivers

Hi Steve,

Thanks for your reply! Very much appreciated.

> From: Steven Rostedt <rostedt@...dmis.org>
> Sent: 15 October 2025 16:08
> To: Fabrizio Castro <fabrizio.castro.jz@...esas.com>
> Cc: davem@...emloft.net; edumazet@...gle.com; kuba@...nel.org; pabeni@...hat.com; andrew@...n.ch;
> bigeasy@...utronix.de; clrkwllms@...nel.org; netdev@...r.kernel.org; linux-rt-devel@...ts.linux.dev
> Subject: Re: Query about the impact of using CONFIG_PREEMPT_RT on locking mechanisms within networking
> drivers
> 
> On Wed, 15 Oct 2025 11:24:35 +0000
> Fabrizio Castro <fabrizio.castro.jz@...esas.com> wrote:
> 
> > Dear All,
> >
> > We have recently started debugging some issues that only show up
> > with the kernel built with CONFIG_PREEMPT_RT=y, and we have noticed
> > some differences w.r.t. the non-RT version.
> >
> > One of the major differences that we have noticed is that spin locks
> > basically become rtmutexes with the RT kernel, whereas they are mapped
> > to raw spin locks in non-RT kernels.
> >
> > When is using raw spin locks directly in networking drivers considered
> > acceptable (if ever)?
> >
> > Thank you for taking the time for reading this email, comments welcome.
> 
> The reason for the spin locks conversion to mutexes is simply to allow for
> more preemption. A raw spin lock can not be preempted. If a lock is held
> for more than a microsecond, you can consider it too long. There's a few

That actually gives us a good way of gauging when holding a lock is not
appropriate. Thanks for this.

> places that may hold locks longer (like the scheduler) but there's no
> choice.
> 
> To allow spin locks to become mutexes, interrupts are also converted into
> threads (including softirqs). There are also "local locks" that are used
> for places that need to protect per-cpu data that is usually protected by
> preempt_disable().
> 
> What issues are you having? It's likely that it can be tweaked so that you
> do not have issues with PREEMPT_RT.

The first issue (which is the one that sparked this discussion) has been
addressed by a patch that was sent out today.
While the issue addressed by that patch is not related in any way to locking,
it sparked a series of discussions within my team about locking because when
PREEMPT_RT is used there are cases where the driver gets preempted at
inconvenient times (while holding a spin lock, that gets translated to an
rtmutex with PREEMPT_RT), and the issue itself is fully masked when using
raw spin locks (and that's because the code doesn't get preempted, making
the issue a lot less likely to show up).

The above picked our curiosity, and therefore we had a look at what's
under `drivers/net` and there doesn't seem to be much code using raw spin
locks directly, hence the question.  

Here is a link to the patch I was mentioning (although not relevant to
locking):
https://lore.kernel.org/netdev/20251015150026.117587-4-prabhakar.mahadev-lad.rj@bp.renesas.com/T/#u

Another issue we have seen is around CPU stalling on a couple of drivers
when PREEMPT_RT is enabled:
https://lore.kernel.org/all/CA+V-a8tWytDVmsk-PK23e4gChXH0pMDR9cKc_xEO4WXpNtr3eA@mail.gmail.com/

The above is more luckily related to locking issues, even though we didn't
have the time to dive into it just yet, so we are not 100% sure about what's
happening just yet.

Again, thanks a lot for your answer.

Kind regards,
Fab

> 
> -- Steve