linux-kernel - Re: [RFC RFC PATCH] mm: convert VM flags from macros to enum

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d619784b-b967-4795-aad9-6e79d4191b83@lucifer.local>
Date: Mon, 13 Oct 2025 12:33:27 +0100
From: Lorenzo Stoakes <lorenzo.stoakes@...cle.com>
To: David Hildenbrand <david@...hat.com>
Cc: Ye Liu <ye.liu@...ux.dev>, Andrew Morton <akpm@...ux-foundation.org>,
        Ye Liu <liuye@...inos.cn>, "Liam R. Howlett" <Liam.Howlett@...cle.com>,
        Vlastimil Babka <vbabka@...e.cz>, Mike Rapoport <rppt@...nel.org>,
        Suren Baghdasaryan <surenb@...gle.com>, Michal Hocko <mhocko@...e.com>,
        linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: [RFC RFC PATCH] mm: convert VM flags from macros to enum

On Mon, Oct 13, 2025 at 01:12:20PM +0200, David Hildenbrand wrote:
> On 13.10.25 13:04, Lorenzo Stoakes wrote:
> > On Sat, Oct 11, 2025 at 05:30:52PM +0800, Ye Liu wrote:
> > > From: Ye Liu <liuye@...inos.cn>
> > >
> > > Hello MM maintainers and drgn community,
> > >
> > > This RFC proposes to convert VM_* flags from #define macros to enum
> > > vm_flags. The motivation comes from recent drgn development where we
> > > encountered difficulties in implementing VM flag parsing due to the
> > > current macro-based approach.
> >
> > This isn't going to work sorry, it's not valid to have flag values as an enum
>
> I don't follow, can you elaborate? IIRC, the compiler will use an integer
> type to back the enum that will fit all values.

switch (flags) {
	case VAL1:
	case VAL2:
	etc.
}

Is broken (compiler will say you cover all cases when you don't...)

An enum implies independent values that exhaustively describe all state, however
these flag values are not that - they're intended to be bit fields.

Also we specifically have the vm_flags_t whose type VMA flags should be, right
now it's an integer type - unsigned long - after my changes it'll be an opaque
type that'll blow up expectations of what this change does altogether.

>
> > (they're distinct) and also - importantly - I'm going to be making significant
> > changes to VMA flags soon (to allow us to have arbitrary number of VMA flags on
> > _all_ architectures as recently done with my series doing something similar with
> > mm flags).
>
> I guess this patch should not really make a big difference regarding your
> upcoming plans?

I don't think it's really useful to do this in any way, and as I said, my change
will allow these tools to have a much better solution, which is to reference the
- actually valid as an enum - bit values.

>
> I do hate the enum stuff to make these tools happy, though :)

Yes, if the tools can't handle standard kernel conventions, the problem is not
in the kernel :)


>
> --
> Cheers
>
> David / dhildenb
>

If it makes life easier - I can prioritise doing the 'make VMA flag bit numbers
an enum' patch as an early part of the upcoming series (and in fact likely to be
multiple series given the scope).

Cheers, Lorenzo