linux-kernel - [RFC] SIGOOM proposal

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <CPKFMO8Q6ON2.OEO6RMYUAK76@hardclanz>
Date:   Thu, 05 Jan 2023 20:28:19 +0300
From:   "Arseniy Lesin" <emptiedsoul@...dclanz.org>
To:     <linux-kernel@...r.kernel.org>
Subject: [RFC] SIGOOM proposal

1. Introduction
===================

AFAIK, majority of linux-running machines are configured to overcommit
memory -- so, memory needs of every process are always pleased. However,
in the Out-Of-Memory condition we repay cruely -- by _killing_ the most
memory-hungry process. 

Dealing with OOM was always questionable -- kernel-space OOM-killer
often works as the last resort -- system can hang for a pretty long time
(especially when using swap) before it resolves the OOM condition.

User-space OOM-killers solve this problem _partially_ -- they can _kill_
such processes preventively or even display nice GUI prompt for user.

However, the key problem persists -- we can only _kill_ unaware process,
possibly causing valuable data loss. There is no way to tell process: "You
are causing system OOM, release memory or you will be terminated forcefully"!

2. Proposal
==================

2.1. The SIGOOM Signal
------------------

I propose the addition of new signal: SIGOOM (Out-Of-Memory SIGnal) 

This signal is intended to be sent to the most memory-hungry process(es)
in order to give process a chance to release memory used for
non-valuable data (for example, browser can unload tabs, that are
currently not in use, assuming tabs are not separate processes) or to
write down valuable data and exit gracefully (for example, some
graphical editor).

Some applications can even set up a poll for OOM event by using signalfd

Default action: 	IGNORE
Proposed senders: 	kernel- and user-space OOM-killers

The technical detail of this addition is a bit unpleasant: there is
actually no room for new signals! 

Numbers 1-31 are already assigned, every signal with number > SIGRTMIN 
(currently 32) is considered realtime and queued accordingly. 

Adding SIGOOM as signal #32 by shifting SIGRTMIN to 33 can do a trick,
but this will almost certainly break compatibility (namely, with glibc
threading)

I propose adding SIGOOM as signal #65 (after SIGRTMAX), but we should
clarify some checks in kernel/signal.c (possibly in other places too,
where signal number is tested against being realtime) and possibly add a
such-like macro:

#define SIG_IS_REALTIME(signum) (((signum) > SIGRTMIN) && ((signum) < SIGRTMAX))

I expect your comments on this topic very much, thanks in advance.

2.2. Adjusting kernel oom-killer to use SIGOOM
----------------------------------------------

Since we now have a way to inform process of it's memory utilization we
can try to send process SIGOOM signal first (if process set up handler
or poll for it) and only then kill it.

=============

I will try to prepare a patchset for kernel in next couple of weeks.
Also going to create some patches for user-space oom-killers (sd-oomd,
meta's oomd) and propose a new system call for those (it is not as
important, so i decided not to include it for this RFC).

I invite all interested to discuss this RFC here on list, or you can
catch me on #linux IRC channel (libera network) tonight (nick:
emptiedsoul)

THX for reading, and again, thanks in advance for your comments.