linux-kernel - Re: [Bug 67651] Bisected: Lots of fragmented mmaps cause gimp to fail in 3.12 after exceeding vm_max_map

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140123121555.GV1574@moon>
Date:	Thu, 23 Jan 2014 16:15:55 +0400
From:	Cyrill Gorcunov <gorcunov@...il.com>
To:	Mel Gorman <mgorman@...e.de>,
	Pavel Emelyanov <xemul@...allels.com>,
	Andrew Morton <akpm@...ux-foundation.org>
Cc:	gnome@...t.net, drawoc@...krefraction.com,
	alan@...rguk.ukuu.org.uk, linux-mm@...ck.org,
	linux-kernel@...r.kernel.org, bugzilla-daemon@...zilla.kernel.org
Subject: Re: [Bug 67651] Bisected: Lots of fragmented mmaps cause gimp to
 fail in 3.12 after exceeding vm_max_map_count

On Thu, Jan 23, 2014 at 02:36:06PM +0400, Cyrill Gorcunov wrote:
> > The test case passes with this patch applied to 3.13 so that appears
> > to confirm that this is related to VM_SOFTDIRTY preventing merges.
> > Unfortunately I did not have slabinfo tracking enabled to double check
> > the number of vm_area_structs in teh system.
> 
> Thanks a lot, Mel! I'm testing the patch as well (manually though :).
> I'll send the final fix today.

The patch below should fix the problem. I would really appreaciate
some additional testing.
---
From: Cyrill Gorcunov <gorcunov@...il.com>
Subject: [PATCH] mm: Ignore VM_SOFTDIRTY on VMA merging

VM_SOFTDIRTY bit affects vma merge routine: if two VMAs has all
bits in vm_flags matched except dirty bit the kernel can't longer
merge them and this forces the kernel to generate new VMAs instead.

It finally may lead to the situation when userspace application
reaches vm.max_map_count limit and get crashed in worse case

 | (gimp:11768): GLib-ERROR **: gmem.c:110: failed to allocate 4096 bytes
 |
 | (file-tiff-load:12038): LibGimpBase-WARNING **: file-tiff-load: gimp_wire_read(): error
 | xinit: connection to X server lost
 |
 | waiting for X server to shut down
 | /usr/lib64/gimp/2.0/plug-ins/file-tiff-load terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup
 | /usr/lib64/gimp/2.0/plug-ins/script-fu terminated: Hangup

https://bugzilla.kernel.org/show_bug.cgi?id=67651
https://bugzilla.gnome.org/show_bug.cgi?id=719619#c0

Initial problem came from missed VM_SOFTDIRTY in do_brk() routine
but even if we would set up VM_SOFTDIRTY here, there is still a way to
prevent VMAs from merging: one can call

 | echo 4 > /proc/$PID/clear_refs

and clear all VM_SOFTDIRTY over all VMAs presented in memory map,
then new do_brk() will try to extend old VMA and finds that dirty
bit doesn't match thus new VMA will be generated.

As discussed to Pavel, the right approach should be to ignore
VM_SOFTDIRTY bit when we're trying to merge VMAs and if merged
successed we mark extended VMA with dirty bit.

Reported-by: Mel Gorman <mgorman@...e.de>
Signed-off-by: Cyrill Gorcunov <gorcunov@...nvz.org>
CC: Pavel Emelyanov <xemul@...allels.com>
CC: Andrew Morton <akpm@...ux-foundation.org>
---
 mm/mmap.c |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

Index: linux-2.6.git/mm/mmap.c
===================================================================
--- linux-2.6.git.orig/mm/mmap.c
+++ linux-2.6.git/mm/mmap.c
@@ -893,7 +893,15 @@ again:			remove_next = 1 + (end > next->
 static inline int is_mergeable_vma(struct vm_area_struct *vma,
 			struct file *file, unsigned long vm_flags)
 {
-	if (vma->vm_flags ^ vm_flags)
+	/*
+	 * VM_SOFTDIRTY should not prevent from VMA merging, if we
+	 * match the flags but dirty bit -- the caller should mark
+	 * merged VMA as dirty. If dirty bit won't be excluded from
+	 * comparison, we increase pressue on the memory system forcing
+	 * the kernel to generate new VMAs when old one could be extended
+	 * instead.
+	 */
+	if ((vma->vm_flags ^ vm_flags) & VM_SOFTDIRTY)
 		return 0;
 	if (vma->vm_file != file)
 		return 0;
@@ -1038,6 +1046,8 @@ struct vm_area_struct *vma_merge(struct
 				end, prev->vm_pgoff, NULL);
 		if (err)
 			return NULL;
+		else
+			prev->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(prev);
 		return prev;
 	}
@@ -1057,6 +1067,8 @@ struct vm_area_struct *vma_merge(struct
 				next->vm_pgoff - pglen, NULL);
 		if (err)
 			return NULL;
+		else
+			area->vm_flags |= VM_SOFTDIRTY;
 		khugepaged_enter_vma_merge(area);
 		return area;
 	}
@@ -1082,7 +1094,7 @@ static int anon_vma_compatible(struct vm
 	return a->vm_end == b->vm_start &&
 		mpol_equal(vma_policy(a), vma_policy(b)) &&
 		a->vm_file == b->vm_file &&
-		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC)) &&
+		!((a->vm_flags ^ b->vm_flags) & ~(VM_READ|VM_WRITE|VM_EXEC|VM_SOFTDIRTY)) &&
 		b->vm_pgoff == a->vm_pgoff + ((b->vm_start - a->vm_start) >> PAGE_SHIFT);
 }
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/