linux-kernel - Re: Regression in performance when using PREEMPT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20260112123356.5bc120b6@gandalf.local.home>
Date: Mon, 12 Jan 2026 12:33:56 -0500
From: Steven Rostedt <rostedt@...dmis.org>
To: Sebastian Andrzej Siewior <bigeasy@...utronix.de>
Cc: LKML <linux-kernel@...r.kernel.org>, linux-rt-devel@...ts.linux.dev,
 liangjlee@...gle.com
Subject: Re: Regression in performance when using PREEMPT_RT

On Mon, 12 Jan 2026 16:27:02 +0100
Sebastian Andrzej Siewior <bigeasy@...utronix.de> wrote:


> If I remember correctly, it completes in the context the
> threaded-handler. It only does the remote-IPI thingy if the queue is
> assigned to a different CPU than the CPU where it completes the request.
> In your case it seems that either the device has multiple queues
> configured and just one interrupt or the queue is configured to a
> different CPU than the interrupt.
> 
> If you have multiple queues but just one interrupt then the lack
> distributing the load is unfortunate. If the queue has been moved to a
> BIG CPU then I suggest to move the IRQ to a BIG CPU, too.
> 
> If you ignore the statement and allow the remote-IPI to kick the softirq
> then the request ends up in ksoftirqd on the remote CPU probably
> accompanied by a warning. Here it runs as SCHED_OTHER and competes for

I'm not sure it triggers a warning. Jack, have you seen that when you
removed the force_irqthreads() check?

> CPU resources with any other task on that CPU which different than
> softirq on !RT where it has to wait until other hardirqs complete.
> The other thing is that if something "else" is busy, say a
> threaded-interrupt then it will pickup this request (before ksoftirqd
> had the chance). The result is that handler now does the I/O at its end
> instead ksoftirqd. If the interrupt is important and has a higher
> priority then the average MAX_RT_PRIO / 2 then this block I/O might
> disrupt its schedule.

I'm not sure what the affect of that would be.

> 
> I think moving the interrupt to a BIG CPU (as the block queue), if
> possible, would be the easiest thing to do.

We have done that. It appears that the hardware simply picks the first CPU
it can use and *always* uses that. By setting it to a big core, it did get
some improvement but the problem is still that the softirq only runs on
that CPU.

> If the hardware restricts it, I would suggest having a dedicated
> SCHED_FIFO thread for its duty would be better than an anonymous
> catch-all ksoftirqd.

I think the solution may be to give up on PREEMPT_RT if that's the case,
unless there's a non PREEMPT_RT reason to make that change.

To give you an idea of what the issue is here, it is the distribution of the
block softirq (even when the irq itself is always coming in on a single
CPU).

 cat /proc/softirqs | grep -i block

non-rt:
	BLOCK:          6          0          0          0          1          7     162986     164750

RT unmodified:
	BLOCK:     329875          0          0          0          0          0          0          0

RT without the forced_irqthreads check:
	BLOCK:          0          0          0          0         11         15     164116     163619

-- Steve