linux-kernel - Re: [BUG] mm/cgroupv2: memory.min may lead to an OOM error

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2527d5a4-de1f-4c93-b7ee-fdd6fbe2a6f0@kernel.org>
Date: Thu, 1 Aug 2024 12:35:27 +0200
From: "Vlastimil Babka (SUSE)" <vbabka@...nel.org>
To: Lance Yang <ioworker0@...il.com>, akpm@...ux-foundation.org
Cc: 21cnbao@...il.com, ryan.roberts@....com, david@...hat.com,
 shy828301@...il.com, ziy@...dia.com, libang.li@...group.com,
 baolin.wang@...ux.alibaba.com, linux-kernel@...r.kernel.org,
 linux-mm@...ck.org, Johannes Weiner <hannes@...xchg.org>,
 Michal Hocko <mhocko@...nel.org>, Roman Gushchin <roman.gushchin@...ux.dev>,
 Shakeel Butt <shakeel.butt@...ux.dev>, Muchun Song <muchun.song@...ux.dev>,
 Cgroups <cgroups@...r.kernel.org>
Subject: Re: [BUG] mm/cgroupv2: memory.min may lead to an OOM error

On 8/1/24 06:54, Lance Yang wrote:
> Hi all,
> 
> It's possible to encounter an OOM error if both parent and child cgroups are
> configured such that memory.min and memory.max are set to the same values, as
> is practice in Kubernetes.

Is it a practice in Kubernetes since forever or a recent one? Did it work
differently before?

> Hmm... I'm not sure that whether this behavior is a bug or an expected aspect of
> the kernel design.

Hmm I'm not a memcg expert, so I cc'd some.

> To reproduce the bug, we can follow these command-based steps:
> 
> 1. Check Kernel Version and OS release:
>     
>     ```
>     $ uname -r
>     6.10.0-rc5+

Were older kernels behaving the same?

Anyway memory.min documentations says "Hard memory protection. If the memory
usage of a cgroup is within its effective min boundary, the cgroup’s memory
won’t be reclaimed under any conditions. If there is no unprotected
reclaimable memory available, OOM killer is invoked."

So to my non-expert opinion this behavior seems valid. if you set min to the
same value as max and then reach the max, you effectively don't allow any
reclaim, so the memcg OOM kill is the only option AFAICS?

>     $ cat /etc/os-release
>     PRETTY_NAME="Ubuntu 24.04 LTS"
>     NAME="Ubuntu"
>     VERSION_ID="24.04"
>     VERSION="24.04 LTS (Noble Numbat)"
>     VERSION_CODENAME=noble
>     ID=ubuntu
>     ID_LIKE=debian
>     HOME_URL="<https://www.ubuntu.com/>"
>     SUPPORT_URL="<https://help.ubuntu.com/>"
>     BUG_REPORT_URL="<https://bugs.launchpad.net/ubuntu/>"
>     PRIVACY_POLICY_URL="<https://www.ubuntu.com/legal/terms-and-policies/privacy-policy>"
>     UBUNTU_CODENAME=noble
>     LOGO=ubuntu-logo
>     
>     ```
>     
> 2. Navigate to the cgroup v2 filesystem, create a test cgroup, and set memory settings:
>     
>     ```
>     $ cd /sys/fs/cgroup/
>     $ stat -fc %T /sys/fs/cgroup
>     cgroup2fs
>     $ mkdir test
>     $ echo "+memory" > cgroup.subtree_control
>     $ mkdir test/test-child
>     $ echo 1073741824 > memory.max
>     $ echo 1073741824 > memory.min
>     $ cat memory.max
>     1073741824
>     $ cat memory.min
>     1073741824
>     $ cat memory.low
>     0
>     $ cat memory.high
>     max
>     ```
>     
> 3. Set up and check memory settings in the child cgroup:
>     
>     ```
>     $ cd test-child
>     $ echo 1073741824 > memory.max
>     $ echo 1073741824 > memory.min
>     $ cat memory.max
>     1073741824
>     $ cat memory.min
>     1073741824
>     $ cat memory.low
>     0
>     $ cat memory.high
>     max
>     ```
>     
> 4. Add process to the child cgroup and verify:
>     
>     ```
>     $ echo $$ > cgroup.procs
>     $ cat cgroup.procs
>     1131
>     1320
>     $ ps -ef|grep 1131
>     root        1131    1014  0 10:45 pts/0    00:00:00 -bash
>     root        1321    1131 99 11:06 pts/0    00:00:00 ps -ef
>     root        1322    1131  0 11:06 pts/0    00:00:00 grep --color=auto 1131
>     ```
>     
> 5. Attempt to create a large file using dd and observe the process being killed:
>     
>     ```
>     $ dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200
>     Killed
>     ```
>     
> 6. Check kernel messages related to the OOM event:
>     
>     ```
>     $ dmesg
>     ...
>     [ 1341.112388] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/test,task_memcg=/test/test-child,task=dd,pid=1324,uid=0
>     [ 1341.112418] Memory cgroup out of memory: Killed process 1324 (dd) total-vm:15548kB, anon-rss:10240kB, file-rss:1764kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0
>     ```
>     
> 7. Reduce the `memory.min` setting in the child cgroup and attempt the same large file creation, and then this issue is resolved.
>     
>     ```
>     # echo 107374182 > memory.min
>     # dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200
>     200+0 records in
>     200+0 records out
>     2097152000 bytes (2.1 GB, 2.0 GiB) copied, 1.8713 s, 1.1 GB/s
>     ```
> 
> Thanks,
> Lance
>