linux-kernel - Re: [PATCH] Fix OOPS in mmap_region() when merging adjacent VM

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 2 Feb 2009 22:35:26 +0000
From:	Mel Gorman <mel@....ul.ie>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Hugh Dickins <hugh@...itas.com>,
	Lee Schermerhorn <Lee.Schermerhorn@...com>,
	Greg KH <gregkh@...e.de>,
	Maksim Yevmenkin <maksim.yevmenkin@...il.com>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	Nick Piggin <npiggin@...e.de>,
	Andrew Morton <akpm@...ux-foundation.org>,
	will@...wder-design.com, Rik van Riel <riel@...hat.com>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	Mikos Szeredi <miklos@...redi.hu>,
	Andy Whitcroft <apw@...onical.com>
Subject: Re: [PATCH] Fix OOPS in mmap_region() when merging adjacent
	VM_LOCKED file segments

On Mon, Feb 02, 2009 at 02:12:30PM -0800, Linus Torvalds wrote:
> 
> 
> On Mon, 2 Feb 2009, Mel Gorman wrote:
> > 
> > Lets say someone does the following
> > 
> > 1. mmap(PROT_READ, MAP_PRIVATE) on a hugetlbfs file
> > 	VM_ACCOUNT is not set for hugetlbfs
> > 	VM_NORESERVE is not set because MAP_NORESERVE was not there
> 
> But isn't this exactly the thing that we have that odd "accountable" flag 
> for, and we do the whole
> 
> 	if (!accountable)
> 		vm_flags |= VM_NORESERVE;
> 
> in mmap_region() for?
> 
> So VM_NORESERVE _will_ be set.
> 

Then it's getting unconditionally set which breaks the hugetlb accounting
for reserving hugepages. See mm/hugetlb.c#decrement_hugepage_resv_vma() and
mm/hugetlb.c#hugetlb_reserve_pages() which depend on VM_NORESERVE being set
or not set depending on MAP_NORESERVE, not whether the core VM is accounting
it or not.

This is why I tried replacing 

 	if (!accountable)
 		vm_flags |= VM_NORESERVE;

with

        if ((flags & MAP_NORESERVE) && should_overcommit(file))
                vm_flags |= VM_NORESERVE;

and

static inline int should_overcommit(struct file *file)
{
        /* Check if the sysctl allows overcommit */
        if (sysctl_overcommit_memory != OVERCOMMIT_NEVER)
                return 1;

        /* hugetlbfs does its own accounting */
        if (file && is_file_hugepages(file))
                return 1;

        return 0;
}

in the first patch I mailed out.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/