| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <da53a828-137a-4efc-a192-a2b49a06d050@linux.dev>
Date: Fri, 22 Aug 2025 23:18:25 +0800
From: Lance Yang <lance.yang@...ux.dev>
To: Geert Uytterhoeven <geert@...ux-m68k.org>, senozhatsky@...omium.org,
mhiramat@...nel.org
Cc: akpm@...ux-foundation.org, will@...nel.org, peterz@...radead.org,
mingo@...hat.com, longman@...hat.com, anna.schumaker@...cle.com,
boqun.feng@...il.com, joel.granados@...nel.org, kent.overstreet@...ux.dev,
leonylgao@...cent.com, linux-kernel@...r.kernel.org, rostedt@...dmis.org,
tfiga@...omium.org, amaindex@...look.com, jstultz@...gle.com,
Mingzhe Yang <mingzhe.yang@...com>, Eero Tamminen <oak@...sinkinet.fi>,
linux-m68k <linux-m68k@...ts.linux-m68k.org>,
Lance Yang <ioworker0@...il.com>
Subject: Re: [PATCH v5 2/3] hung_task: show the blocker task if the task is
hung on semaphore
Hi Geert,
Thanks a lot for bisecting and reporting!
On 2025/8/22 15:38, Geert Uytterhoeven wrote:
> Hi Lance,
>
> (this time the right email thread, I hope ;-)
>
> On Mon, 14 Apr 2025 at 17:23, Lance Yang <ioworker0@...il.com> wrote:
>> Inspired by mutex blocker tracking[1], this patch makes a trade-off to
>> balance the overhead and utility of the hung task detector.
>>
>> Unlike mutexes, semaphores lack explicit ownership tracking, making it
>> challenging to identify the root cause of hangs. To address this, we
>> introduce a last_holder field to the semaphore structure, which is
>> updated when a task successfully calls down() and cleared during up().
>>
>> The assumption is that if a task is blocked on a semaphore, the holders
>> must not have released it. While this does not guarantee that the last
>> holder is one of the current blockers, it likely provides a practical hint
>> for diagnosing semaphore-related stalls.
>>
[...]
>
> Thanks for your patch, which is now commit 194a9b9e843b4077
> ("hung_task: show the blocker task if the task is hung on
> semaphore") in v6.16-rc1.
>
> Eero reported [1] two WARNINGS seen with v6.16 on emulated Atari.
> I managed to reproduce it on ARAnyM using the provided config (it does
> not happen with atari_defconfig), and bisected it to this commit:
The two warnings are directly related, and the first one
is the root cause, IIUC.
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 39 at include/linux/hung_task.h:48
The first warning at hung_task.h:48 is triggered because
WARN_ON_ONCE(lock_ptr & BLOCKER_TYPE_MASK) check fails.
static inline void hung_task_set_blocker(void *lock, unsigned long type)
{
unsigned long lock_ptr = (unsigned long)lock;
WARN_ON_ONCE(!lock_ptr);
WARN_ON_ONCE(READ_ONCE(current->blocker));
/*
* If the lock pointer matches the BLOCKER_TYPE_MASK, return
* without writing anything.
*/
if (WARN_ON_ONCE(lock_ptr & BLOCKER_TYPE_MASK)) <- here
return;
This logic assumes the lock pointer is sufficiently aligned,
allowing the lower bits to be used for the lock type. But it
appears we are being passed an unaligned lock pointer,
unfortunately.
WRITE_ONCE(current->blocker, lock_ptr | type);
}
Because the check fails, hung_task_set_blocker() returns
early without setting current->blocker, which directly
leads to the second warning at hung_task.h:56
> __down_common+0x13a/0x1be
> CPU: 0 UID: 0 PID: 39 Comm: getty Not tainted
> 6.15.0-rc6hatari-00018-g194a9b9e843b #1986 NONE
> Stack from 01633d00:
> 01633d00 00366e9e 00366e9e 00000000 002c9762 00360cb5 01633d24 0000873e
> 00366e9e 01633d40 0002e0d4 00360cb5 00000030 00000009 0039c79a 00061408
> 01633d78 000028e0 00360cb5 00000030 002c9762 00000009 00000000 00000000
> 7fffffff 00000002 1185d266 01633eb0 01326c58 00000080 01633dc0 002c9762
> 00360cb5 00000030 00000009 00000000 00002014 01326c00 1185d266 01633eb0
> 002c93ea 00053d60 00061408 01326c58 0038db90 0038db90 01633e32 01633fb8
> Call Trace: [<002c9762>] __down_common+0x13a/0x1be
> [<0000873e>] dump_stack+0x10/0x16
> [<0002e0d4>] __warn+0x7a/0xbc
> [<00061408>] msleep+0x0/0x2c
> [<000028e0>] warn_slowpath_fmt+0x42/0x62
> [<002c9762>] __down_common+0x13a/0x1be
> [<002c9762>] __down_common+0x13a/0x1be
> [<00002014>] arch_local_irq_enable+0xe/0x22
> [<002c93ea>] mutex_lock+0x0/0x28
> [<00053d60>] other_cpu_in_panic+0x0/0x26
> [<00061408>] msleep+0x0/0x2c
> [<002c97fc>] __down+0x16/0x1e
> [<002c9832>] down+0x2e/0x30
> [<00053dac>] console_lock+0x26/0x4c
> [<001aae4e>] do_con_write+0x3a/0x16d4
> [<002c93ea>] mutex_lock+0x0/0x28
> [<0004fa70>] __add_wait_queue+0x3a/0x6a
> [<001ac520>] con_write+0x1a/0x30
> [<0019cafa>] n_tty_write+0x2c6/0x35e
> [<00199456>] signal_pending+0x0/0x26
> [<000aba2a>] __kvmalloc_node_noprof+0x3a/0x114
> [<00004cc0>] io_uring_try_cancel_requests+0x98/0x318
> [<0004fb2e>] woken_wake_function+0x0/0x24
> [<0019a180>] file_tty_write.isra.0+0x144/0x1b8
> [<0019a206>] tty_write+0x12/0x16
> [<000b97c2>] vfs_write+0xec/0x148
> [<00028000>] fp_getdest+0x1b8/0x224
> [<00010000>] g_trace+0x16/0x28
> [<000b9916>] ksys_write+0x54/0x8a
> [<000b9962>] sys_write+0x16/0x1a
> [<000093da>] syscall+0x8/0xc
> [<0000c001>] arch_dma_prep_coherent+0x51/0x58
>
> ---[ end trace 0000000000000000 ]---
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 39 at include/linux/hung_task.h:56
Later, when hung_task_clear_blocker() is called, its
WARN_ON_ONCE triggers because it finds current->blocker
is still zero from the earlier failure.
static inline void hung_task_clear_blocker(void)
{
WARN_ON_ONCE(!READ_ONCE(current->blocker)); <- here
WRITE_ONCE(current->blocker, 0UL);
}
So, the unaligned lock pointer appears to be the root cause,
breaking the assumptions of the blocker tracking mechanism.
> __down_common+0x17a/0x1be
> CPU: 0 UID: 0 PID: 39 Comm: getty Tainted: G W
> 6.15.0-rc6hatari-00018-g194a9b9e843b #1986 NONE
> Tainted: [W]=WARN
> Stack from 01633d00:
> 01633d00 00366e9e 00366e9e 00000000 002c97a2 00360cb5 01633d24 0000873e
> 00366e9e 01633d40 0002e0d4 00360cb5 00000038 00000009 0039c79a 01633db2
> 01633d78 000028e0 00360cb5 00000038 002c97a2 00000009 00000000 00000000
> 00000000 00000002 00000000 00000000 01326c58 0039c79a 01633dc0 002c97a2
> 00360cb5 00000038 00000009 00000000 00002014 01326c00 1185d266 01633eb0
> 002c93ea 00053d60 00061408 01326c58 00380000 01000000 01220162 b64001b8
> Call Trace: [<002c97a2>] __down_common+0x17a/0x1be
> [<0000873e>] dump_stack+0x10/0x16
> [<0002e0d4>] __warn+0x7a/0xbc
> [<000028e0>] warn_slowpath_fmt+0x42/0x62
> [<002c97a2>] __down_common+0x17a/0x1be
> [<002c97a2>] __down_common+0x17a/0x1be
> [<00002014>] arch_local_irq_enable+0xe/0x22
> [<002c93ea>] mutex_lock+0x0/0x28
> [<00053d60>] other_cpu_in_panic+0x0/0x26
> [<00061408>] msleep+0x0/0x2c
> [<002c97fc>] __down+0x16/0x1e
> [<002c9832>] down+0x2e/0x30
> [<00053dac>] console_lock+0x26/0x4c
> [<001aae4e>] do_con_write+0x3a/0x16d4
> [<002c93ea>] mutex_lock+0x0/0x28
> [<0004fa70>] __add_wait_queue+0x3a/0x6a
> [<001ac520>] con_write+0x1a/0x30
> [<0019cafa>] n_tty_write+0x2c6/0x35e
> [<00199456>] signal_pending+0x0/0x26
> [<000aba2a>] __kvmalloc_node_noprof+0x3a/0x114
> [<00004cc0>] io_uring_try_cancel_requests+0x98/0x318
> [<0004fb2e>] woken_wake_function+0x0/0x24
> [<0019a180>] file_tty_write.isra.0+0x144/0x1b8
> [<0019a206>] tty_write+0x12/0x16
> [<000b97c2>] vfs_write+0xec/0x148
> [<00028000>] fp_getdest+0x1b8/0x224
> [<00010000>] g_trace+0x16/0x28
> [<000b9916>] ksys_write+0x54/0x8a
> [<000b9962>] sys_write+0x16/0x1a
> [<000093da>] syscall+0x8/0xc
> [<0000c001>] arch_dma_prep_coherent+0x51/0x58
>
> ---[ end trace 0000000000000000 ]---
>
> It still happens on v6.17-rc2. Reverting commits 77da18de55ac6417
> ("hung_task: extend hung task blocker tracking to rwsems") and
> 194a9b9e843b4077 ("hung_task: show the blocker task if the task is
> hung on semaphore") fixes the issue for me.
Thanks! I'm looking into it.
Lance
>
> Thanks!
>
> [1] "v6.16 console issues on Atari Falcon"
> https://lore.kernel.org/all/92518308-c763-4591-96ef-6b38c5d8f434@helsinkinet.fi
>
> Gr{oetje,eeting}s,
>
> Geert
>
Powered by blists - more mailing lists