linux-kernel - Re: Latency spikes on V6.15.1 Preempt RT and maybe related to intel? IGB

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250617091621.IZh5OyYh@linutronix.de>
Date: Tue, 17 Jun 2025 11:16:21 +0200
From: Sebastian Andrzej Siewior <bigeasy@...utronix.de>
To: Marc Strämke <marc.straemke@...ropuls.de>
Cc: linux-kernel@...r.kernel.org, linux-rt-users@...r.kernel.org
Subject: Re: Latency spikes on V6.15.1 Preempt RT and maybe related to intel?
 IGB

On 2025-06-16 17:10:33 [+0200], Marc Strämke wrote:
> Hi Sebastian,
Hi Marc,

> I am still trying to figure that puzzle out: Please see the following
> function trace snippet:
> 
> ip-690530  [000] ..... 178636.460435: rtnl_is_locked <-__dev_change_flags
> ip-690530  [000] ..... 178636.460435: __local_bh_disable_ip <-__dev_change_flags
> ip-690530  [000] ..... 178636.460435: migrate_disable <-__local_bh_disable_ip
> ip-690530  [000] ...1. 178636.460435: preempt_disable: caller=__local_bh_disable_ip+0x76/0xe0 parent=__local_bh_disable_ip+0x76/0xe0
> ip-690530  [000] ...11 178636.460435: preempt_enable: caller=__local_bh_disable_ip+0x76/0xe0 parent=__local_bh_disable_ip+0x76/0xe0
> ip-690530  [000] ....1 178636.460435: rt_spin_lock <-__local_bh_disable_ip
> ip-690530  [000] ....1 178636.460436: __rcu_read_lock <-rt_spin_lock
> ip-690530  [000] ....1 178636.460436: migrate_disable <-__local_bh_disable_ip
> ip-690530  [000] ....2 178636.460436: __rcu_read_lock <-__local_bh_disable_ip
> ip-690530  [000] b...2 178636.460436: rt_spin_lock <-__dev_change_flags
> ip-690530  [000] b...2 178636.460436: __rcu_read_lock <-rt_spin_lock
> ip-690530  [000] b...2 178636.460436: migrate_disable <-__dev_change_flags
> ip-690530  [000] b...3 178636.460436: __dev_set_rx_mode <-__dev_change_flags
> ip-690530  [000] b...3 178636.460437: igb_set_rx_mode <-__dev_change_flags
> 
> It is my understanding that __local_bh_disable_ip called from
> netif_addr_lock_bh seems to be the origin of my latency.

How so?

> What i do not understand is, even if the bottom halves are disabled.
> Shouldn't I see the interrupt arriving in the trace?
Yes.

> If i understood it correctly, your talk "Unblocking the softirq lock for
> PREEMPT_RT" during the plumbers conference 2023 is exactly about that case,
> right?
No.

> Probably fixing this issue is out of my abilities for now ;-) The wast
> variety of locking concepts inside the kernel is quite intimidating for a
> newcomer in kernel land...

If you would have only the scheduler events enabled and you would see
that "ip-690530" is doing something, then an interrupt fires, that
interrupts wakes a thread, CPU switches to that thread and that thread
does sched_pi_setprio() and switches back to "ip-690530" until it is
done _then_ it would be what I said on plumbers in 2023.

Your trace snippet above is short latency wise. It covers just 2us.
From that it all looks good. If you go further down in your trace,
there has to be a spike between igb_update_mc_addr_list() and
asm_sysvec_apic_timer_interrupt().

> Regards,
> 
> 
> Marc

Sebastian