linux-kernel - Re: [PATCH v3 1/5] mm: fix off-by-one error in VMA count limit checks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <kqxweui7gdnnwu6gucnn4ez7cwq57xih43p7dgvqa7tvfc6vjg@b4alxiizejsc>
Date: Tue, 14 Oct 2025 13:51:31 -0400
From: "Liam R. Howlett" <Liam.Howlett@...cle.com>
To: Hugh Dickins <hughd@...gle.com>
Cc: Kalesh Singh <kaleshsingh@...gle.com>, akpm@...ux-foundation.org,
        minchan@...nel.org, lorenzo.stoakes@...cle.com, david@...hat.com,
        rppt@...nel.org, pfalcato@...e.de, kernel-team@...roid.com,
        android-mm@...gle.com, stable@...r.kernel.org,
        SeongJae Park <sj@...nel.org>,
        Alexander Viro <viro@...iv.linux.org.uk>,
        Christian Brauner <brauner@...nel.org>, Jan Kara <jack@...e.cz>,
        Kees Cook <kees@...nel.org>, Vlastimil Babka <vbabka@...e.cz>,
        Suren Baghdasaryan <surenb@...gle.com>, Michal Hocko <mhocko@...e.com>,
        Jann Horn <jannh@...gle.com>, Steven Rostedt <rostedt@...dmis.org>,
        Masami Hiramatsu <mhiramat@...nel.org>,
        Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        Ingo Molnar <mingo@...hat.com>, Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Valentin Schneider <vschneid@...hat.com>,
        Shuah Khan <shuah@...nel.org>, linux-kernel@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, linux-mm@...ck.org,
        linux-trace-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v3 1/5] mm: fix off-by-one error in VMA count limit checks

* Hugh Dickins <hughd@...gle.com> [251014 02:28]:
> On Mon, 13 Oct 2025, Kalesh Singh wrote:
> 
> > The VMA count limit check in do_mmap() and do_brk_flags() uses a
> > strict inequality (>), which allows a process's VMA count to exceed
> > the configured sysctl_max_map_count limit by one.

...

> >  	/* Too many mappings? */
> > -	if (mm->map_count > sysctl_max_map_count)
> > +	if (mm->map_count >= sysctl_max_map_count)
> >  		return -ENOMEM;
> >  
> >  	/*
> > diff --git a/mm/vma.c b/mm/vma.c
> > index a2e1ae954662..fba68f13e628 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -2797,7 +2797,7 @@ int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma,
> >  	if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT))
> >  		return -ENOMEM;
> >  
> > -	if (mm->map_count > sysctl_max_map_count)
> > +	if (mm->map_count >= sysctl_max_map_count)
> >  		return -ENOMEM;
> >  
> >  	if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT))
> > -- 
> > 2.51.0.760.g7b8bcc2412-goog
> 
> Sorry for letting you go so far before speaking up (I had to test what
> I believed to be true, and had hoped that meanwhile one of your many
> illustrious reviewers would say so first, but no): it's a NAK from me.
> 
> These are not off-by-ones: at the point of these checks, it is not
> known whether an additional map/vma will have to be added, or the
> addition will be merged into an existing map/vma.  So the checks
> err on the lenient side, letting you get perhaps one more than the
> sysctl said, but not allowing any more than that.
> 
> Which is all that matters, isn't it? Limiting unrestrained growth.
> 
> In this patch you're proposing to change it from erring on the
> lenient side to erring on the strict side - prohibiting merges
> at the limit which have been allowed for many years.
> 
> Whatever one thinks about the merits of erring on the lenient versus
> erring on the strict side, I see no reason to make this change now,
> and most certainly not with a Fixes Cc: stable. There is no danger
> in the current behaviour; there is danger in prohibiting what was
> allowed before.

Thanks Hugh.

I'm left wondering if the issue is that we are checking in the wrong
location.  That is, should we be checking so early in the process or
later when we know the count adjustment?

But then again, later we may be in mid-operation and find out we're out
of room.  Other places are even more lenient and allow us to exceed the
count for a potential limited time, and we really don't know what's
going to happen until we examine what already exists.. So it seems like
a lot of effort to avoid one extra vma.

> 
> As to the remainder of your series: I have to commend you for doing
> a thorough and well-presented job, but I cannot myself see the point in
> changing 21 files for what almost amounts to a max_map_count subsystem.
> I call it misdirected effort, not at all to my taste, which prefers the
> straightforward checks already there; but accept that my taste may be
> out of fashion, so won't stand in the way if others think it worthwhile.

I'm not sure which way I favour, it does seem like a large change to
avoid an issue that never existed.

In either case, it seems like a good idea to adjust the comments to
state that the count may not change.

Thanks,
Liam