linux-kernel - [PATCH] hugetlbfs: Kill applications that use MAP_NORESERVE with SIGBUS instead of OOM-killer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100420174407.GA30306@csn.ul.ie>
Date:	Tue, 20 Apr 2010 18:44:07 +0100
From:	Mel Gorman <mel@....ul.ie>
To:	Andrew Morton <akpm@...ux-foundation.org>
Cc:	Lee Schermerhorn <lee.schermerhorn@...com>,
	David Rientjes <rientjes@...gle.com>,
	Andi Kleen <andi@...stfloor.org>, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org
Subject: [PATCH] hugetlbfs: Kill applications that use MAP_NORESERVE with
	SIGBUS instead of OOM-killer

Ordinarily, application using hugetlbfs will create mappings with
reserves. For shared mappings, these pages are reserved before mmap()
returns success and for private mappings, the caller process is
guaranteed and a child process that cannot get the pages gets killed
with sigbus.

An application that uses MAP_NORESERVE gets no reservations and mmap()
will always succeed at the risk the page will not be available at fault
time. This might be used for example on very large sparse mappings where the
developer is confident the necessary huge pages exist to satisfy all faults
even though the whole mapping cannot be backed by huge pages.  Unfortunately,
if an allocation does fail, VM_FAULT_OOM is returned to the fault handler
which proceeds to trigger the OOM-killer. This is unhelpful.

This patch alters hugetlbfs to kill a process that uses MAP_NORESERVE
where huge pages were not available with SIGBUS instead of triggering
the OOM killer.

This patch if accepted should also be considered a -stable candidate.

Signed-off-by: Mel Gorman <mel@....ul.ie>
---
 mm/hugetlb.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6034dc9..af2d907 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1038,7 +1038,7 @@ static struct page *alloc_huge_page(struct vm_area_struct *vma,
 		page = alloc_buddy_huge_page(h, vma, addr);
 		if (!page) {
 			hugetlb_put_quota(inode->i_mapping, chg);
-			return ERR_PTR(-VM_FAULT_OOM);
+			return ERR_PTR(-VM_FAULT_SIGBUS);
 		}
 	}

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/