linux-kernel - Re: [Bug 67651] Bisected: Lots of fragmented mmaps cause gimp to fail in 3.12 after exceeding vm_max_map

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Y5EM+zlgFvoGBG5X@grain>
Date:   Thu, 8 Dec 2022 01:00:27 +0300
From:   Cyrill Gorcunov <gorcunov@...il.com>
To:     Mel Gorman <mgorman@...e.de>
Cc:     Pavel Emelyanov <xemul@...allels.com>,
        Andrew Morton <akpm@...ux-foundation.org>, gnome@...t.net,
        drawoc@...krefraction.com, alan@...rguk.ukuu.org.uk,
        linux-mm@...ck.org, linux-kernel@...r.kernel.org,
        bugzilla-daemon@...zilla.kernel.org
Subject: Re: [Bug 67651] Bisected: Lots of fragmented mmaps cause gimp to
 fail in 3.12 after exceeding vm_max_map_count

On Thu, Jan 23, 2014 at 10:30:44AM +0000, Mel Gorman wrote:
> 
> The test case passes with this patch applied to 3.13 so that appears
> to confirm that this is related to VM_SOFTDIRTY preventing merges.
> Unfortunately I did not have slabinfo tracking enabled to double check
> the number of vm_area_structs in teh system.

Hi Mel! I'm really really sorry for replying that late, somehow missed the former
bug report (note the bugzilla date message as Jan 23, 2014) so no wonder. Actually
I don't understand yet how SOFTDIRTY can prevent merging. When VMAs are to merge
we explicitly ignore softdirty flag

/*
 * If the vma has a ->close operation then the driver probably needs to release
 * per-vma resources, so we don't attempt to merge those.
 */
static inline int is_mergeable_vma(struct vm_area_struct *vma,
				struct file *file, unsigned long vm_flags,
				struct vm_userfaultfd_ctx vm_userfaultfd_ctx,
				struct anon_vma_name *anon_name)
{
	/*
	 * VM_SOFTDIRTY should not prevent from VMA merging, if we
	 * match the flags but dirty bit -- the caller should mark
	 * merged VMA as dirty. If dirty bit won't be excluded from
	 * comparison, we increase pressure on the memory system forcing
	 * the kernel to generate new VMAs when old one could be
	 * extended instead.
	 */
	if ((vma->vm_flags ^ vm_flags) & ~VM_SOFTDIRTY)
		return 0;
	...
}

so the softdirty flag is not preventing VMAs from being merged.

	Cyrill