linux-kernel - Re: [PATCH 3/4] mm: prevent KSM from completely breaking VMA merging

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <3e2d3bbb-8610-41d3-9aee-5a7bba3f2ce8@redhat.com>
Date: Mon, 19 May 2025 20:04:22 +0200
From: David Hildenbrand <david@...hat.com>
To: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
 Andrew Morton <akpm@...ux-foundation.org>
Cc: Alexander Viro <viro@...iv.linux.org.uk>,
 Christian Brauner <brauner@...nel.org>, Jan Kara <jack@...e.cz>,
 "Liam R . Howlett" <Liam.Howlett@...cle.com>,
 Vlastimil Babka <vbabka@...e.cz>, Jann Horn <jannh@...gle.com>,
 Pedro Falcato <pfalcato@...e.de>, Xu Xin <xu.xin16@....com.cn>,
 Chengming Zhou <chengming.zhou@...ux.dev>, linux-mm@...ck.org,
 linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH 3/4] mm: prevent KSM from completely breaking VMA merging

>> +/*
>> + * Are we guaranteed no driver can change state such as to preclude KSM merging?
>> + * If so, let's set the KSM mergeable flag early so we don't break VMA merging.
>> + *
>> + * This is applicable when PR_SET_MEMORY_MERGE has been set on the mm_struct via
>> + * prctl() causing newly mapped VMAs to have the KSM mergeable VMA flag set.
>> + *
>> + * If this is not the case, then we set the flag after considering mergeability,
>> + * which will prevent mergeability as, when PR_SET_MEMORY_MERGE is set, a new
>> + * VMA will not have the KSM mergeability VMA flag set, but all other VMAs will,
>> + * preventing any merge.
> 
> Hmmm, so an ordinary MAP_PRIVATE of any file (executable etc.) will get
> VM_MERGEABLE set but not be able to merge?
> 
> Probably these are not often expected to be merged ...
> 
> Preventing merging should really only happen because of VMA flags that
> are getting set: VM_PFNMAP, VM_MIXEDMAP, VM_DONTEXPAND, VM_IO.
> 
> 
> I am not 100% sure why we bail out on special mappings: all we have to
> do is reliably identify anon pages, and we should be able to do that.
> 
> GUP does currently refuses any VM_PFNMAP | VM_IO, and KSM uses GUP,
> which might need a tweak then (maybe the solution could be to ... not
> use GUP but a folio_walk).

Oh, someone called "David" already did that. Nice :)

So we *should* be able to drop

* VM_PFNMAP: we correctly identify CoWed pages
* VM_MIXEDMAP: we correctly identify CoWed pages
* VM_IO: should not affect CoWed pages
* VM_DONTEXPAND: no idea why that should even matter here

-- 
Cheers,

David / dhildenb