linux-kernel - [QUESTION] oom killed the key system process triggered by a bad process alloc memory with MAP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 1 Nov 2021 16:05:50 +0800
From:   Yongqiang Liu <liuyongqiang13@...wei.com>
To:     <mhocko@...nel.org>
CC:     <rientjes@...gle.com>, <linux-mm@...ck.org>,
        <linux-kernel@...r.kernel.org>,
        <penguin-kernel@...ove.sakura.ne.jp>,
        "Wangkefeng (OS Kernel Lab)" <wangkefeng.wang@...wei.com>
Subject: [QUESTION] oom killed the key system process triggered by a bad
 process alloc memory with MAP_LOCKED

Hi,

I was just wondering the process of  the patch "oom reaper: handle 
mlocked pages".

links:

https://lore.kernel.org/linux-mm/1454505240-23446-3-git-send-email-mhocko@kernel.org/

https://lore.kernel.org/linux-mm/20160223132157.GD14178@dhcp22.suse.cz/


In linux master, we found that when we start a process and alloc large 
memory with MAP_LOCKED, the oom will triggered and kill some system 
process such as sshd ,rsyslog etc.

...

[   45.110665] Out of memory: Killed process 2551 (oom) 
total-vm:1035324kB, anon-rss:1028196kB, file-rss:704kB, shmem-rss:0kB, 
UID:0 pgtables:2056kB oom_score_adj:1000
[   45.115303] Out of memory: Killed process 2554 (oom) 
total-vm:1025084kB, anon-rss:1015596kB, file-rss:388kB, shmem-rss:0kB, 
UID:0 pgtables:2032kB oom_score_adj:1000
[   45.115685] Out of memory: Killed process 2553 (oom) 
total-vm:953404kB, anon-rss:947748kB, file-rss:388kB, shmem-rss:0kB, 
UID:0 pgtables:1896kB oom_score_adj:1000
[   45.116031] Out of memory: Killed process 2552 (oom) 
total-vm:789564kB, anon-rss:783272kB, file-rss:388kB, shmem-rss:0kB, 
UID:0 pgtables:1576kB oom_score_adj:1000
[   45.117199] Out of memory: Killed process 2523 (sshd) 
total-vm:77052kB, anon-rss:804kB, file-rss:4672kB, shmem-rss:4kB, UID:0 
pgtables:196kB oom_score_adj:0
[   45.120936] Out of memory: Killed process 2526 (bash) 
total-vm:17792kB, anon-rss:1180kB, file-rss:2868kB, shmem-rss:0kB, UID:0 
pgtables:76kB oom_score_adj:0

...

the process demo is:

...

#define ALLOC_SIZE (10 * 1024 * 1024)

while (1) {

     addr = mmap(NULL, ALLOC_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | 
MAP_ANONYMOUS | MAP_LOCKED, -1, 0);

     if (addr != MAP_FAILED)

         memset(addr, 1, ALLOC_SIZE);

}

...

And we found that when the oom_reaper is done but the memory is still high:

[   45.115685] Out of memory: Killed process 2553 (oom) 
total-vm:953404kB, anon-rss:947748kB, file-rss:388kB, shmem-rss:0kB, 
UID:0 pgtables:1896kB oom_score_adj:1000
[   45.115739] oom_reaper: reaped process 2553 (oom), now 
anon-rss:947708kB, file-rss:0kB, shmem-rss:0kB

This is because the bad proccess which recieved SIGKILL is unlocking the 
mem to exit which needs more time. And the next oom is triggered to kill 
the other system process.

Kind regards,

Yongqiang Liu