linux-kernel - Re: [PATCH v9 1/2] mm/oom_kill: Thaw the entire OOM victim process

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <y7hdpmq6smlv2eztltwilmxcvalax63y66ttbckewa6mrk45yl@nu2kv3vtbsxv>
Date: Thu, 11 Sep 2025 16:55:55 -0700
From: Shakeel Butt <shakeel.butt@...ux.dev>
To: zhongjinji <zhongjinji@...or.com>
Cc: mhocko@...e.com, rientjes@...gle.com, akpm@...ux-foundation.org, 
	tglx@...utronix.de, liam.howlett@...cle.com, lorenzo.stoakes@...cle.com, 
	surenb@...gle.com, lenb@...nel.org, rafael@...nel.org, pavel@...nel.org, 
	linux-mm@...ck.org, linux-pm@...r.kernel.org, linux-kernel@...r.kernel.org, 
	liulu.liu@...or.com, feng.han@...or.com
Subject: Re: [PATCH v9 1/2] mm/oom_kill: Thaw the entire OOM victim process

On Wed, Sep 10, 2025 at 10:37:25PM +0800, zhongjinji wrote:
> OOM killer is a mechanism that selects and kills processes when the system
> runs out of memory to reclaim resources and keep the system stable. But the
> oom victim cannot terminate on its own when it is frozen, even if the OOM
> victim task is thawed through __thaw_task(). This is because __thaw_task() can
> only thaw a single OOM victim thread, and cannot thaw the entire OOM victim
> process.
> 
> Also, freezing_slow_path() decides whether a task is an OOM victim by checking
> the task's TIF_MEMDIE flag. When a task is thawed, the freezer bypasses PM
> freezing and cgroup freezing states to thaw it. But TIF_MEMDIE is not a thread
> group shared flag, and only one thread is marked with TIF_MEMDIE. If other
> threads are thawed, they may still remain frozen due to PM freezing and cgroup
> freezing states.
> 
> To solve this, thaw_process() is introduced to thaw all threads of the victim,
> ensuring every thread in the victim process can be thawed. The freezer uses
> tsk_is_oom_victim() to determine whether a task is an OOM victim, because
> tsk->signal->oom_mm is data shared by all threads. This allows all victim threads
> to rely on it to be thawed.
> 
> This change will thaw the entire victim process when OOM occurs,
> ensuring that the oom victim can terminate on its own.
> 
> Signed-off-by: zhongjinji <zhongjinji@...or.com>
> 
> Acked-by: Michal Hocko <mhocko@...e.com>

Acked-by: Shakeel Butt <shakeel.butt@...ux.dev>