linux-kernel - Re: [PATCH 0/4] mm: ksm: prevent KSM from entirely breaking VMA merging

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <bd7a2e05-af48-4d0f-b546-3d053690eba4@lucifer.local>
Date: Mon, 19 May 2025 12:56:03 +0100
From: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
To: David Hildenbrand <david@...hat.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>,
        Alexander Viro <viro@...iv.linux.org.uk>,
        Christian Brauner <brauner@...nel.org>, Jan Kara <jack@...e.cz>,
        "Liam R . Howlett" <Liam.Howlett@...cle.com>,
        Vlastimil Babka <vbabka@...e.cz>, Jann Horn <jannh@...gle.com>,
        Pedro Falcato <pfalcato@...e.de>, Xu Xin <xu.xin16@....com.cn>,
        Chengming Zhou <chengming.zhou@...ux.dev>, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH 0/4] mm: ksm: prevent KSM from entirely breaking VMA
 merging

On Mon, May 19, 2025 at 01:53:43PM +0200, David Hildenbrand wrote:
> On 19.05.25 10:51, Lorenzo Stoakes wrote:
> > When KSM-by-default is established using prctl(PR_SET_MEMORY_MERGE), this
> > defaults all newly mapped VMAs to having VM_MERGEABLE set, and thus makes
> > them available to KSM for samepage merging. It also sets VM_MERGEABLE in
> > all existing VMAs.
> >
> > However this causes an issue upon mapping of new VMAs - the initial flags
> > will never have VM_MERGEABLE set when attempting a merge with adjacent VMAs
> > (this is set later in the mmap() logic), and adjacent VMAs will ALWAYS have
> > VM_MERGEABLE set.
>
> Just to clarify, you mean that VM_MERGEABLE is set later, during
> __mmap_new_vma()->ksm_add_vma()->__ksm_add_vma(), and we are already past
> vma_merge_new_range(), correct?

Yes.

The self test asserts this is in fact the behaviour, and if you run it in a
kernel without this patchset you can observe it in action.

>
> --
> Cheers,
>
> David / dhildenb
>