linux-kernel - Re: [PATCH 1/1] sched/numa: Fix memory leak due to the overwritten vma->numab

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1794be3c-358c-4cdc-a43d-a1f841d91ef7@amd.com>
Date: Sat, 9 Nov 2024 09:33:57 +0530
From: Raghavendra K T <raghavendra.kt@....com>
To: Adrian Huang <adrianhuang0701@...il.com>, Ingo Molnar <mingo@...hat.com>,
 Peter Zijlstra <peterz@...radead.org>, Juri Lelli <juri.lelli@...hat.com>,
 Vincent Guittot <vincent.guittot@...aro.org>,
 Andrew Morton <akpm@...ux-foundation.org>
Cc: Dietmar Eggemann <dietmar.eggemann@....com>,
 Steven Rostedt <rostedt@...dmis.org>, Ben Segall <bsegall@...gle.com>,
 Mel Gorman <mgorman@...e.de>, Valentin Schneider <vschneid@...hat.com>,
 linux-mm@...ck.org, linux-kernel@...r.kernel.org,
 Adrian Huang <ahuang12@...ovo.com>, Jiwei Sun <sunjw10@...ovo.com>
Subject: Re: [PATCH 1/1] sched/numa: Fix memory leak due to the overwritten
 vma->numab_state

On 11/8/2024 7:01 PM, Adrian Huang wrote:
> From: Adrian Huang <ahuang12@...ovo.com>
> 
> [Problem Description]
> When running the hackbench program of LTP, the following memory leak is
> reported by kmemleak.
> 
>    # /opt/ltp/testcases/bin/hackbench 20 thread 1000
>    Running with 20*40 (== 800) tasks.
> 
>    # dmesg | grep kmemleak
>    ...
>    kmemleak: 480 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
>    kmemleak: 665 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
> 
>    # cat /sys/kernel/debug/kmemleak
>    unreferenced object 0xffff888cd8ca2c40 (size 64):
>      comm "hackbench", pid 17142, jiffies 4299780315
>      hex dump (first 32 bytes):
>        ac 74 49 00 01 00 00 00 4c 84 49 00 01 00 00 00  .tI.....L.I.....
>        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
>      backtrace (crc bff18fd4):
>        [<ffffffff81419a89>] __kmalloc_cache_noprof+0x2f9/0x3f0
>        [<ffffffff8113f715>] task_numa_work+0x725/0xa00
>        [<ffffffff8110f878>] task_work_run+0x58/0x90
>        [<ffffffff81ddd9f8>] syscall_exit_to_user_mode+0x1c8/0x1e0
>        [<ffffffff81dd78d5>] do_syscall_64+0x85/0x150
>        [<ffffffff81e0012b>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
>    ...
> 
>    This issue can be consistently reproduced on three different servers:
>      * a 448-core server
>      * a 256-core server
>      * a 192-core server
> 
> [Root Cause]
> Since multiple threads are created by the hackbench program (along with
> the command argument 'thread'), a shared vma might be accessed by two or
> more cores simultaneously. When two or more cores observe that
> vma->numab_state is NULL at the same time, vma->numab_state will be
> overwritten.
> 

Thanks for reporting.

IIRC, This is not the entire scenario. Chunk above the vma->numab code
ideally ensures, only one thread descend down to scan the VMA's in a
single 'numa_scan_period'

     migrate = mm->numa_next_scan;
         if (time_before(now, migrate))
                 return;
         next_scan = now + msecs_to_jiffies(p->numa_scan_period);
         if (!try_cmpxchg(&mm->numa_next_scan, &migrate, next_scan))
                 return;

However since there are 800 threads, I see there might be an opportunity
for another thread to enter in the next 'numa_scan_period' while
we have not gotten till numab_state allocation.

There should be simpler ways to overcome like Vlastimil already pointed
in the other thread, and having lock is an overkill.

for e.g.,
numab_state = kzalloc(..)

if we see that some other thread able to successfully assign
vma->numab_state with their allocation (with cmpxchg), simply
free your allocation.

Can you please check if my understanding is correct?

Thanks
- Raghu

[...]