linux-kernel - Re: Latency spikes on V6.15.1 Preempt RT and maybe related to intel? IGB

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d72bc0b1-7898-4513-9b50-60b56112c8e1@eltropuls.de>
Date: Mon, 16 Jun 2025 17:10:33 +0200
From: Marc Strämke <marc.straemke@...ropuls.de>
To: Sebastian Andrzej Siewior <bigeasy@...utronix.de>
Cc: linux-kernel@...r.kernel.org, linux-rt-users@...r.kernel.org
Subject: Re: Latency spikes on V6.15.1 Preempt RT and maybe related to intel?
 IGB

Hi Sebastian,

On 14.06.25 10:52, Marc Strämke wrote:
>
> I do not really understand where the IRQ/Preemption disabling is 
> happening. What would the next thing be to do? Function (graph?) 
> tracing on all the functions visible in the backtrace?
>
>
> I tried to look at the event race output starting with the call to 
> igb_set_rx_mode. I have attached the trace with all events and a 
> function filter on igb on only the cpu executing ip.  I cannot 
> understand what is happening between timestasmp 700.149995 and the IRQ 
> disable event on 700.150795....
>
>
> Thanks for your help,
>
> Marc
>
>
I am still trying to figure that puzzle out: Please see the following 
function trace snippet:

ip-690530  [000] ..... 178636.460435: rtnl_is_locked <-__dev_change_flags
ip-690530  [000] ..... 178636.460435: __local_bh_disable_ip 
<-__dev_change_flags
ip-690530  [000] ..... 178636.460435: migrate_disable 
<-__local_bh_disable_ip
ip-690530  [000] ...1. 178636.460435: preempt_disable: 
caller=__local_bh_disable_ip+0x76/0xe0 
parent=__local_bh_disable_ip+0x76/0xe0
ip-690530  [000] ...11 178636.460435: preempt_enable: 
caller=__local_bh_disable_ip+0x76/0xe0 
parent=__local_bh_disable_ip+0x76/0xe0
ip-690530  [000] ....1 178636.460435: rt_spin_lock <-__local_bh_disable_ip
ip-690530  [000] ....1 178636.460436: __rcu_read_lock <-rt_spin_lock
ip-690530  [000] ....1 178636.460436: migrate_disable 
<-__local_bh_disable_ip
ip-690530  [000] ....2 178636.460436: __rcu_read_lock 
<-__local_bh_disable_ip
ip-690530  [000] b...2 178636.460436: rt_spin_lock <-__dev_change_flags
ip-690530  [000] b...2 178636.460436: __rcu_read_lock <-rt_spin_lock
ip-690530  [000] b...2 178636.460436: migrate_disable <-__dev_change_flags
ip-690530  [000] b...3 178636.460436: __dev_set_rx_mode <-__dev_change_flags
ip-690530  [000] b...3 178636.460437: igb_set_rx_mode <-__dev_change_flags

It is my understanding that __local_bh_disable_ip called from 
netif_addr_lock_bh seems to be the origin of my latency.

What i do not understand is, even if the bottom halves are disabled. 
Shouldn't I see the interrupt arriving in the trace?

If i understood it correctly, your talk "Unblocking the softirq lock for 
PREEMPT_RT" during the plumbers conference 2023 is exactly about that 
case, right?

Probably fixing this issue is out of my abilities for now ;-) The wast 
variety of locking concepts inside the kernel is quite intimidating for 
a newcomer in kernel land...

Regards,

Marc