linux-kernel - Re: [PATCH 0/4] mm: ksm: prevent KSM from entirely breaking VMA merging

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c9de8232-b938-477a-a6b5-271c27ed85fe@lucifer.local>
Date: Wed, 21 May 2025 19:23:17 +0100
From: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
To: Andrew Morton <akpm@...ux-foundation.org>
Cc: Alexander Viro <viro@...iv.linux.org.uk>,
        Christian Brauner <brauner@...nel.org>, Jan Kara <jack@...e.cz>,
        "Liam R . Howlett" <Liam.Howlett@...cle.com>,
        Vlastimil Babka <vbabka@...e.cz>, Jann Horn <jannh@...gle.com>,
        Pedro Falcato <pfalcato@...e.de>, David Hildenbrand <david@...hat.com>,
        Xu Xin <xu.xin16@....com.cn>,
        Chengming Zhou <chengming.zhou@...ux.dev>, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org,
        Stefan Roesch <shr@...kernel.io>
Subject: Re: [PATCH 0/4] mm: ksm: prevent KSM from entirely breaking VMA
 merging

And in the annals of 'hit send then immediately notice mistake' we have
entry number 1,379 :P

This subject should read [PATCH v2 0/4] obviously, like the patches that
reply to it in-thread.

Apologies for that!

Other than that, series should be fine :>)

Cheers, Lorenzo

On Wed, May 21, 2025 at 07:20:27PM +0100, Lorenzo Stoakes wrote:
> When KSM-by-default is established using prctl(PR_SET_MEMORY_MERGE), this
> defaults all newly mapped VMAs to having VM_MERGEABLE set, and thus makes
> them available to KSM for samepage merging. It also sets VM_MERGEABLE in
> all existing VMAs.
>
> However this causes an issue upon mapping of new VMAs - the initial flags
> will never have VM_MERGEABLE set when attempting a merge with adjacent VMAs
> (this is set later in the mmap() logic), and adjacent VMAs will ALWAYS have
> VM_MERGEABLE set.
>
> This renders literally all VMAs in the virtual address space unmergeable.
>
> To avoid this, this series performs the check for PR_SET_MEMORY_MERGE far
> earlier in the mmap() logic, prior to the merge being attempted.
>
> However we run into a complexity with the depreciated .mmap() callback - if
> a driver hooks this, it might change flags thus adjusting KSM merge
> eligibility.
>
> This isn't a problem for brk(), where the VMA must be anonymous. However
> for mmap() we are conservative - if the VMA is anonymous then we can always
> proceed, however if not, we permit only shmem mappings and drivers which
> implement .mmap_prepare().
>
> If we can't be sure of the driver changing things, then we maintain the
> same behaviour of performing the KSM check later in the mmap() logic (and
> thus losing VMA mergeability).
>
> Since the .mmap_prepare() hook is invoked prior to the KSM check, this
> means we can always perform the KSM check early if it is present. Over time
> as drivers are converted, we can do away with the later check altogether.
>
> A great many use-cases for this logic will use anonymous or shmem memory at
> any rate, as KSM is not supported for the page cache, and the driver
> outliers in question are MAP_PRIVATE mappings of these files.
>
> So this change should already cover the majority of actual KSM use-cases.
>
> v2:
> * Removed unnecessary ret local variable in ksm_vma_flags() as per David.
> * added Stefan Roesch in cc and added Fixes tag as per Andrew, David.
> * Propagated tags (thanks everyone!)
> * Removed unnecessary !CONFIG_KSM ksm_add_vma() stub from
>   include/linux/ksm.h.
> * Added missing !CONFIG_KSM ksm_vma_flags() stub in
>   include/linux/ksm.h.
> * After discussion with David, I've decided to defer removing the
>   VM_SPECIAL case for KSM, we can address this in a follow-up series.
> * Expanded 3/4 commit message to reference KSM eligibility vs. merging and
>   referenced future plans to permit KSM for VM_SPECIAL VMAs.
>
> v1:
> https://lore.kernel.org/all/cover.1747431920.git.lorenzo.stoakes@oracle.com/
>
> Lorenzo Stoakes (4):
>   mm: ksm: have KSM VMA checks not require a VMA pointer
>   mm: ksm: refer to special VMAs via VM_SPECIAL in ksm_compatible()
>   mm: prevent KSM from completely breaking VMA merging
>   tools/testing/selftests: add VMA merge tests for KSM merge
>
>  include/linux/fs.h                 |  7 ++-
>  include/linux/ksm.h                |  8 +--
>  mm/ksm.c                           | 49 ++++++++++++-------
>  mm/vma.c                           | 49 ++++++++++++++++++-
>  tools/testing/selftests/mm/merge.c | 78 ++++++++++++++++++++++++++++++
>  5 files changed, 167 insertions(+), 24 deletions(-)
>
> --
> 2.49.0
>