linux-kernel - Re: Arithmetic overflow in may_expand

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20091019075350.GA1769@cmpxchg.org>
Date:	Mon, 19 Oct 2009 09:53:50 +0200
From:	Johannes Weiner <hannes@...xchg.org>
To:	Earl Chew <earl_chew@...lent.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Arithmetic overflow in may_expand_vm()

Hi,

On Thu, Oct 15, 2009 at 10:24:51AM -0700, Earl Chew wrote:
> This code currently reads:
> 
> >int may_expand_vm(struct mm_struct *mm, unsigned long npages)
> >{
> >        unsigned long cur = mm->total_vm;       /* pages */
> >        unsigned long lim;
> >
> >        lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT;
> >
> >        if (cur + npages > lim)
> >                return 0;
> >        return 1;
> >}
> 
> If npages is stupendously large, the failure predicate may
> return a false negative due to (cur + npages) overflowing and
> wrapping.

Can this really happen?

npages always originates in a value of byte granularity, giving a
theoretical maximum of ~0UL >> PAGE_SHIFT (checking for more than the
number of addressable bytes just makes no sense).

And mm->total_vm is always PAGE_SIZE times smaller than total user
address space (which in turn is always less than ~0UL).

So I can not see this overflow being possible with PAGE_SHIFT > 0.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/