netdev - Re: deadlocks on pernet_ops

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+FuTSfnOzbZrztVYX26M6xAd5Y-hP0=Ek7svJpDUSLKApK0aQ@mail.gmail.com>
Date: Wed, 1 Oct 2025 15:33:56 -0400
From: Willem de Bruijn <willemb@...gle.com>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Kuniyuki Iwashima <kuniyu@...gle.com>, edumazet@...gle.com, fw@...len.de, 
	netdev@...r.kernel.org, pabeni@...hat.com
Subject: Re: deadlocks on pernet_ops_rwsem

On Wed, Oct 1, 2025 at 3:26 PM Jakub Kicinski <kuba@...nel.org> wrote:
>
> On Wed,  1 Oct 2025 18:50:22 +0000 Kuniyuki Iwashima wrote:
> > From: Jakub Kicinski <kuba@...nel.org>
> > Date: Wed, 1 Oct 2025 10:22:23 -0700
> > > To be clear -- AFAICT lockdep misses this.
> > >
> > > The splat is from the "stuck task" checker.
> > >
> > > 2 min wait to load a module during test init would definitely be a sign
> > > of something going sideways.. but I think it's worse than that, these
> > > time out completely and we kill the VM. I think the modprobe is truly
> > > stuck here.
> > >
> > > In one of the splats lockdep was able to say:
> > >
> > > [ 4302.448228][   T44] INFO: task modprobe:31634 <writer> blocked on an rw-semaphore likely owned by task kworker/u16:0:12 <reader>
> > >
> > > but most are more useless:
> > >
> > > [ 4671.090728][   T44] INFO: task modprobe:2342 is blocked on an rw-semaphore, but the owner is not found.
> > >
> > > (?!?)
> >
> > Even when it caught the possible owner, lockdep seems confused :/
> >
> >
> > [ 4302.448228][   T44] INFO: task modprobe:31634 <writer> blocked on an rw-semaphore likely owned by task kworker/u16:0:12 <reader>
> >
> > modprobe:31634 seems to be blocked by kworker/u16:0:12,
> >
> >
> > [ 4302.449035][   T44] task:kworker/u16:0   state:R  running task     stack:26368 pid:12    tgid:12    ppid:2      task_flags:0x4208060 flags:0x00004000
> > [ 4302.449872][   T44] Workqueue: netns cleanup_net
> > ...
> > [ 4302.460889][   T44] Showing all locks held in the system:
> > [ 4302.461368][   T44] 4 locks held by kworker/u16:0/12:
> >
> > but no lock shows up here for kworker/u16:0/12,
> >
> >
> > [ 4302.461597][   T44] 2 locks held by kworker/u18:0/36:
> > [ 4302.461926][   T44]  #0: ffff8880010d9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7e5/0x1650
> > [ 4302.462429][   T44]  #1: ffffc9000028fd40 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0xded/0x1650
> > [ 4302.463011][   T44] 1 lock held by khungtaskd/44:
> > [ 4302.463261][   T44]  #0: ffffffffb7b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260
> > [ 4302.463717][   T44] 1 lock held by modprobe/31634:
> > [ 4302.463982][   T44]  #0: ffffffffb8270430 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40
> >
> > and modprobe/31634 is holding pernet_ops_rwsem ???
> >
> >
> > Was there any update on packages (especially qemu?) used by
> > CI around 2025-09-18 ?
>
> No updates according to the logs. First hit was on Thursday so I thought
> maybe it came from Linus. But looking at the branches we fast forwarded
> 2025-09-18--21-00 and there were 2 hits earlier that day (2025-09-18--03-00,
> 2025-09-18--15-00)

Is there a good SHA1 around the time of the earliest report?

There do not seem to be any recent changes to rwsem, let alone
specifically to pernet_ops_rwsem.