linux-kernel - [PATCH] mm/mmap: Add case 9 in vma

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20240218085028.3294332-1-yajun.deng@linux.dev>
Date: Sun, 18 Feb 2024 16:50:28 +0800
From: Yajun Deng <yajun.deng@...ux.dev>
To: akpm@...ux-foundation.org
Cc: Liam.Howlett@...cle.com,
	linux-mm@...ck.org,
	linux-kernel@...r.kernel.org,
	Yajun Deng <yajun.deng@...ux.dev>
Subject: [PATCH] mm/mmap: Add case 9 in vma_merge()

If the prev vma exists and the end is less than the end of prev, we
can return NULL immediately. This reduces unnecessary operations.

Signed-off-by: Yajun Deng <yajun.deng@...ux.dev>
---
 mm/mmap.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 8f176027583c..b738849321c0 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -827,7 +827,7 @@ can_vma_merge_after(struct vm_area_struct *vma, unsigned long vm_flags,
  *
  *     ****             ****                   ****
  *    PPPPPPNNNNNN    PPPPPPNNNNNN       PPPPPPCCCCCC
- *    cannot merge    might become       might become
+ *    cannot merge 9  might become       might become
  *                    PPNNNNNNNNNN       PPPPPPPPPPCC
  *    mmap, brk or    case 4 below       case 5 below
  *    mremap move:
@@ -890,6 +890,9 @@ static struct vm_area_struct
 	if (vm_flags & VM_SPECIAL)
 		return NULL;
 
+	if (prev && end < prev->vm_end) /* case 9 */
+		return NULL;
+
 	/* Does the input range span an existing VMA? (cases 5 - 8) */
 	curr = find_vma_intersection(mm, prev ? prev->vm_end : 0, end);
 
-- 
2.25.1