| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <f43ce864-4b4f-4b48-815f-0fa3e338cee0@lucifer.local>
Date: Mon, 14 Oct 2024 10:04:51 +0100
From: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
To: "Liam R. Howlett" <Liam.Howlett@...cle.com>,
Sasha Levin <sashal@...nel.org>,
syzbot <syzbot+39bc767144c55c8db0ea@...kaller.appspotmail.com>,
akpm@...ux-foundation.org, linux-kernel@...r.kernel.org,
linux-mm@...ck.org, syzkaller-bugs@...glegroups.com, vbabka@...e.cz
Subject: Re: [syzbot] [mm?] INFO: task hung in exit_mmap
On Sun, Oct 13, 2024 at 10:22:11PM -0400, Liam R. Howlett wrote:
> * Sasha Levin <sashal@...nel.org> [241013 09:29]:
> > On Thu, Oct 10, 2024 at 04:28:18PM +0100, Lorenzo Stoakes wrote:
> > > On Thu, Oct 10, 2024 at 08:19:28AM -0700, syzbot wrote:
> > > > Hello,
> > > >
> > > > syzbot found the following issue on:
> > > >
> > > > HEAD commit: d3d1556696c1 Merge tag 'mm-hotfixes-stable-2024-10-09-15-4..
> > > > git tree: upstream
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=10416fd0580000
> > > > kernel config: https://syzkaller.appspot.com/x/.config?x=7a3fccdd0bb995
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=39bc767144c55c8db0ea
> > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > > >
> > > > Unfortunately, I don't have any reproducer for this issue yet.
> > > >
> > > > Downloadable assets:
> > > > disk image: https://storage.googleapis.com/syzbot-assets/0600b551e610/disk-d3d15566.raw.xz
> > > > vmlinux: https://storage.googleapis.com/syzbot-assets/d59d43ed3976/vmlinux-d3d15566.xz
> > > > kernel image: https://storage.googleapis.com/syzbot-assets/e686a3e7e0d6/bzImage-d3d15566.xz
> > > >
> > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > Reported-by: syzbot+39bc767144c55c8db0ea@...kaller.appspotmail.com
> > > >
> > > > INFO: task syz.3.917:7739 blocked for more than 146 seconds.
> > > > Not tainted 6.12.0-rc2-syzkaller-00074-gd3d1556696c1 #0
> > > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > task:syz.3.917 state:D stack:23808 pid:7739 tgid:7739 ppid:5232 flags:0x00004000
> > > > Call Trace:
> > > > <TASK>
> > > > context_switch kernel/sched/core.c:5322 [inline]
> > > > __schedule+0x1843/0x4ae0 kernel/sched/core.c:6682
> > > > __schedule_loop kernel/sched/core.c:6759 [inline]
> > > > schedule+0x14b/0x320 kernel/sched/core.c:6774
> > > > schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6831
> > > > rwsem_down_write_slowpath+0xeee/0x13b0 kernel/locking/rwsem.c:1176
> > > > __down_write_common kernel/locking/rwsem.c:1304 [inline]
> > > > __down_write kernel/locking/rwsem.c:1313 [inline]
> > > > down_write+0x1d7/0x220 kernel/locking/rwsem.c:1578
> > > > mmap_write_lock include/linux/mmap_lock.h:106 [inline]
> > > > exit_mmap+0x2bd/0xc40 mm/mmap.c:1872
> > >
> > > Hmm, task freezing up or system becoming unstable/locked up is reminsecent
> > > of the maple tree bug I fixed in [0], which is still in the unstable hotfix
> > > branch.
> > >
> > > This is likely not going to repro as it's quite heisenbug-ish to trigger
> > > and the failures are like this - somewhat disconnected from the cause, so
> > > not sure if there is any case to speed this to Linus's tree.
> > >
> > > On the other hand it's a pretty serious problem for stability and likely to
> > > continue to manifest in nasty ways like this.
> > >
> > > Can't be 100% sure this is the cause, but seems likely.
> > >
> > > [0]:https://lore.kernel.org/linux-mm/48b349a2a0f7c76e18772712d0997a5e12ab0a3b.1728314403.git.lorenzo.stoakes@oracle.com/
> >
> > On my Debian build box, running a 6.1 kernel, I've started hitting a
> > similar issue:
> >
> > Oct 12 17:24:01 debian kernel: INFO: task sed:3557356 blocked for more than 1208 seconds.
> > Oct 12 17:24:01 debian kernel: Not tainted 6.1.0-26-amd64 #1 Debian 6.1.112-1
> > Oct 12 17:24:01 debian kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Oct 12 17:24:01 debian kernel: task:sed state:D stack:0 pid:3557356 ppid:1 flags:0x00000002
> > Oct 12 17:24:01 debian kernel: Call Trace:
> > Oct 12 17:24:01 debian kernel: <TASK>
> > Oct 12 17:24:01 debian kernel: __schedule+0x34d/0x9e0
> > Oct 12 17:24:01 debian kernel: schedule+0x5a/0xd0
> > Oct 12 17:24:01 debian kernel: rwsem_down_write_slowpath+0x311/0x6d0
> > Oct 12 17:24:01 debian kernel: exit_mmap+0xf6/0x2f0
> > Oct 12 17:24:01 debian kernel: __mmput+0x3e/0x130
> > Oct 12 17:24:01 debian kernel: do_exit+0x2fc/0xaf0
> > Oct 12 17:24:01 debian kernel: do_group_exit+0x2d/0x80
> > Oct 12 17:24:01 debian kernel: __x64_sys_exit_group+0x14/0x20
> > Oct 12 17:24:01 debian kernel: do_syscall_64+0x55/0xb0
> > Oct 12 17:24:01 debian kernel: ? do_fault+0x1a4/0x410
> > Oct 12 17:24:01 debian kernel: ? __handle_mm_fault+0x660/0xfa0
> > Oct 12 17:24:01 debian kernel: ? exit_to_user_mode_prepare+0x40/0x1e0
> > Oct 12 17:24:01 debian kernel: ? handle_mm_fault+0xdb/0x2d0
> > Oct 12 17:24:01 debian kernel: ? do_user_addr_fault+0x1b0/0x550
> > Oct 12 17:24:01 debian kernel: ? exit_to_user_mode_prepare+0x40/0x1e0
> > Oct 12 17:24:01 debian kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> > Oct 12 17:24:01 debian kernel: RIP: 0033:0x7f797d75a349
> > Oct 12 17:24:01 debian kernel: RSP: 002b:00007fff37f0d3c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> > Oct 12 17:24:01 debian kernel: RAX: ffffffffffffffda RBX: 00007f797d8549e0 RCX: 00007f797d75a349
> > Oct 12 17:24:01 debian kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> > Oct 12 17:24:01 debian kernel: RBP: 0000000000000000 R08: fffffffffffffe98 R09: 00007fff37f0d2df
> > Oct 12 17:24:01 debian kernel: R10: 00007fff37f0d240 R11: 0000000000000246 R12: 00007f797d8549e0
> > Oct 12 17:24:01 debian kernel: R13: 00007f797d85a2e0 R14: 0000000000000002 R15: 00007f797d85a2c8
> > Oct 12 17:24:01 debian kernel: </TASK>
> >
> > It reproduces fairly easily during a kernel build...
> >
> > It doesn't sound like the same issue you're pointing out, right Lorenzo?
>
> It could be. I suspect there has been a change recently that has
> made the bug possible - although, I've not put effort into finding out
> yet if that is true. If the bug existed for a long time (probably since
> I fixed the live locking issue in 6.4 that was backported), then you
> could be hitting it.
>
> It is a single line fix. If it happens frequently enough, you could try
> it - this fix will go through the backporting route once it lands.
>
> Although, I am not sure it has much to do with the maple tree as I don't
> think anyone should have the mm to take the mmap write lock. If we were
> stuck in the maple tree somehow, the mm wouldn't reach the exit_mmap()
> path - unless I missed something?
I think this is the same bug, as the problem is, once it manifests, the
actual problems it causes are delayed down the line until we hit a
situation that is _caused by_ the bug but somewhat detached from it.
In fact Bert and Mikhail hit the same thing with a process locking up in
this path AND in that a lock is held that really makes no sense.
As Liam says, this is a one-line change [0], could you try taking it and
seeing? It has already been taken by Andrew, but we delay our hotfixes by a
week or two so isn't in -rc yet.
[0]: https://lore.kernel.org/linux-mm/48b349a2a0f7c76e18772712d0997a5e12ab0a3b.1728314403.git.lorenzo.stoakes@oracle.com/
>
> If you can dump the running tasks when you hit it, we could get a clue
> from the (probably numerous) backtraces?
I suggest first trying [0] :) if that doesn't fix things then we will
definitely want to dig in further.
>
> Thanks,
> Liam
Thanks!
Powered by blists - more mailing lists