linux-kernel - Re: [RFC PATCH v3 00/12] Use obj_cgroup APIs to charge the LRU pages

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMZfGtUiKcM8WmP88J3K5edwLhJhsUkAUQo6rnkqx4BBOEY2SA@mail.gmail.com>
Date:   Tue, 18 May 2021 21:49:25 +0800
From:   Muchun Song <songmuchun@...edance.com>
To:     Roman Gushchin <guro@...com>, Johannes Weiner <hannes@...xchg.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Linux Memory Management List <linux-mm@...ck.org>,
        Vladimir Davydov <vdavydov.dev@...il.com>,
        Shakeel Butt <shakeelb@...gle.com>,
        Xiongchun duan <duanxiongchun@...edance.com>,
        fam.zheng@...edance.com, "Singh, Balbir" <bsingharora@...il.com>,
        Yang Shi <shy828301@...il.com>,
        Alex Shi <alex.shi@...ux.alibaba.com>,
        Michal Hocko <mhocko@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [RFC PATCH v3 00/12] Use obj_cgroup APIs to charge the LRU pages

Ping...

Hi Johannes and Roman,

Any suggestions on this patch set?

Thanks.

On Wed, Apr 21, 2021 at 3:01 PM Muchun Song <songmuchun@...edance.com> wrote:
>
> This is v3 based on the top of the series[1] (memcontrol code cleanup and
> simplification). Roman is working on the generalization of obj_cgroup API.
> But before that, hope someone can review this patches for correctness.
>
> Since the following patchsets applied. All the kernel memory are charged
> with the new APIs of obj_cgroup.
>
>         [v17,00/19] The new cgroup slab memory controller[2]
>         [v5,0/7] Use obj_cgroup APIs to charge kmem pages[3]
>
> But user memory allocations (LRU pages) pinning memcgs for a long time -
> it exists at a larger scale and is causing recurring problems in the real
> world: page cache doesn't get reclaimed for a long time, or is used by the
> second, third, fourth, ... instance of the same job that was restarted into
> a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory,
> and make page reclaim very inefficient.
>
> We can convert LRU pages and most other raw memcg pins to the objcg direction
> to fix this problem, and then the LRU pages will not pin the memcgs.
>
> This patchset aims to make the LRU pages to drop the reference to memory
> cgroup by using the APIs of obj_cgroup. Finally, we can see that the number
> of the dying cgroups will not increase if we run the following test script.
>
> ```bash
> #!/bin/bash
>
> cat /proc/cgroups | grep memory
>
> cd /sys/fs/cgroup/memory
>
> for i in range{1..500}
> do
>         mkdir test
>         echo $$ > test/cgroup.procs
>         sleep 60 &
>         echo $$ > cgroup.procs
>         echo `cat test/cgroup.procs` > cgroup.procs
>         rmdir test
> done
>
> cat /proc/cgroups | grep memory
> ```
>
> Thanks.
>
> [1] https://lore.kernel.org/linux-mm/20210417043538.9793-1-songmuchun@bytedance.com/
> [2] https://lore.kernel.org/linux-mm/20200623015846.1141975-1-guro@fb.com/
> [3] https://lore.kernel.org/linux-mm/20210319163821.20704-1-songmuchun@bytedance.com/
>
> Changlogs in RFC v3:
>   1. Drop the code cleanup and simplification patches. Gather those patches
>      into a separate series[1].
>   2. Rework patch #1 suggested by Johannes.
>
> Changlogs in RFC v2:
>   1. Collect Acked-by tags by Johannes. Thanks.
>   2. Rework lruvec_holds_page_lru_lock() suggested by Johannes. Thanks.
>   3. Fix move_pages_to_lru().
>
> Muchun Song (12):
>   mm: memcontrol: move the objcg infrastructure out of CONFIG_MEMCG_KMEM
>   mm: memcontrol: introduce compact_lock_page_lruvec_irqsave
>   mm: memcontrol: make lruvec lock safe when the LRU pages reparented
>   mm: vmscan: rework move_pages_to_lru()
>   mm: thp: introduce lock/unlock_split_queue{_irqsave}()
>   mm: thp: make deferred split queue lock safe when the LRU pages
>     reparented
>   mm: memcontrol: make all the callers of page_memcg() safe
>   mm: memcontrol: introduce memcg_reparent_ops
>   mm: memcontrol: use obj_cgroup APIs to charge the LRU pages
>   mm: memcontrol: rename {un}lock_page_memcg() to {un}lock_page_objcg()
>   mm: lru: add VM_BUG_ON_PAGE to lru maintenance function
>   mm: lru: use lruvec lock to serialize memcg changes
>
>  Documentation/admin-guide/cgroup-v1/memory.rst |   2 +-
>  fs/buffer.c                                    |  13 +-
>  fs/fs-writeback.c                              |  23 +-
>  fs/iomap/buffered-io.c                         |   4 +-
>  include/linux/memcontrol.h                     | 182 ++++----
>  include/linux/mm_inline.h                      |   6 +
>  mm/compaction.c                                |  36 +-
>  mm/filemap.c                                   |   2 +-
>  mm/huge_memory.c                               | 171 ++++++--
>  mm/memcontrol.c                                | 562 ++++++++++++++++++-------
>  mm/migrate.c                                   |   4 +
>  mm/page-writeback.c                            |  24 +-
>  mm/page_io.c                                   |   5 +-
>  mm/rmap.c                                      |  14 +-
>  mm/swap.c                                      |  46 +-
>  mm/vmscan.c                                    |  56 ++-
>  16 files changed, 795 insertions(+), 355 deletions(-)
>
> --
> 2.11.0
>