linux-kernel - [RFC PATCH 0/4] Removing limitations of merging anonymous VMAs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20220218122019.130274-1-matenajakub@gmail.com>
Date:   Fri, 18 Feb 2022 13:20:15 +0100
From:   Jakub Matěna <matenajakub@...il.com>
To:     linux-mm@...ck.org
Cc:     patches@...ts.linux.dev, linux-kernel@...r.kernel.org,
        vbabka@...e.cz, mhocko@...nel.org, mgorman@...hsingularity.net,
        willy@...radead.org, liam.howlett@...cle.com, hughd@...gle.com,
        kirill@...temov.name, riel@...riel.com, rostedt@...dmis.org,
        peterz@...radead.org,
        Jakub Matěna <matenajakub@...il.com>
Subject: [RFC PATCH 0/4] Removing limitations of merging anonymous VMAs

Motivation
In the current kernel it is impossible to merge two anonymous VMAs
if one of them was moved. That is because VMA's page offset is
set according to the virtual address where it was created and in
order to merge two VMA's page offsets need to follow up.
Another problem when merging two VMA's is their anon_vma. In
current kernel these anon_vmas have to be the one and the same.
Otherwise merge is again not allowed.
Missed merge opportunities increase the number of VMAs of a process
and in some cases can cause problems when a max count is reached.

Solution
Following series of these patches solves the first problem with
page offsets by updating them when the VMA is moved to a
different virtual address (patch 2). As for the second
problem merging of VMAs with different anon_vma is allowed
(patch 3). Patch 1 refactors function vma_merge and
makes it easier to understand and also allows relatively
seamless tracing of successful merges introduced by the patch 4.

Limitations
For both problems solution works only for VMAs that do not share
physical pages with other processes (usually child or parent
processes). This is checked by looking at anon_vma of the respective
VMA. The reason why it is not possible or at least not easy to
accomplish is that each physical page has a pointer to anon_vma and
page offset. And when this physical page is shared we cannot simply
change these parameters without affecting all of the VMAs mapping
this physical page. Good thing is that this case amounts only for
about 1-3% of all merges (measured for internet browsing and
compilation use cases) that fail to merge in the current kernel.

This series of patches and documentation of the related code will
be part of my master's thesis.
This patch series is based on tag v5.17-rc4.

Jakub Matěna (4):
  mm: refactor of vma_merge()
  mm: adjust page offset in mremap
  mm: enable merging of VMAs with different anon_vmas
  mm: add tracing for VMA merges

 include/linux/rmap.h        |  17 ++-
 include/trace/events/mmap.h |  55 +++++++++
 mm/internal.h               |  11 ++
 mm/mmap.c                   | 232 ++++++++++++++++++++++++++----------
 mm/rmap.c                   |  40 +++++++
 5 files changed, 290 insertions(+), 65 deletions(-)

-- 
2.34.1