| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <ab29ca4d-a7dc-9115-930d-86c6425e2b9c@bytedance.com>
Date: Fri, 8 Sep 2023 17:58:56 +0800
From: Peng Zhang <zhangpeng.00@...edance.com>
To: "Liam R. Howlett" <Liam.Howlett@...cle.com>,
Peng Zhang <zhangpeng.00@...edance.com>, corbet@....net,
akpm@...ux-foundation.org, willy@...radead.org, brauner@...nel.org,
surenb@...gle.com, michael.christie@...cle.com,
peterz@...radead.org, mathieu.desnoyers@...icios.com,
npiggin@...il.com, avagin@...il.com, linux-mm@...ck.org,
linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-fsdevel@...r.kernel.org
Subject: Re: [PATCH v2 6/6] fork: Use __mt_dup() to duplicate maple tree in
dup_mmap()
在 2023/9/8 04:14, Liam R. Howlett 写道:
> * Peng Zhang <zhangpeng.00@...edance.com> [230830 08:58]:
>> Use __mt_dup() to duplicate the old maple tree in dup_mmap(), and then
>> directly modify the entries of VMAs in the new maple tree, which can
>> get better performance. The optimization effect is proportional to the
>> number of VMAs.
>>
>> There is a "spawn" in byte-unixbench[1], which can be used to test the
>> performance of fork(). I modified it slightly to make it work with
>> different number of VMAs.
>>
>> Below are the test numbers. There are 21 VMAs by default. The first row
>> indicates the number of added VMAs. The following two lines are the
>> number of fork() calls every 10 seconds. These numbers are different
>> from the test results in v1 because this time the benchmark is bound to
>> a CPU. This way the numbers are more stable.
>>
>> Increment of VMAs: 0 100 200 400 800 1600 3200 6400
>> 6.5.0-next-20230829: 111878 75531 53683 35282 20741 11317 6110 3158
>> Apply this patchset: 114531 85420 64541 44592 28660 16371 9038 4831
>> +2.37% +13.09% +20.23% +26.39% +38.18% +44.66% +47.92% +52.98%
>
> Thanks!
>
> Can you include 21 in this table since it's the default?
Maybe I didn't express clearly, "Increment of VMAs" means the number of
VMAs added on the basis of 21 VMAs.
>
>>
>> [1] https://github.com/kdlucas/byte-unixbench/tree/master
>>
>> Signed-off-by: Peng Zhang <zhangpeng.00@...edance.com>
>> ---
>> kernel/fork.c | 34 ++++++++++++++++++++++++++--------
>> mm/mmap.c | 14 ++++++++++++--
>> 2 files changed, 38 insertions(+), 10 deletions(-)
>>
>> diff --git a/kernel/fork.c b/kernel/fork.c
>> index 3b6d20dfb9a8..e6299adefbd8 100644
>> --- a/kernel/fork.c
>> +++ b/kernel/fork.c
>> @@ -650,7 +650,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>> int retval;
>> unsigned long charge = 0;
>> LIST_HEAD(uf);
>> - VMA_ITERATOR(old_vmi, oldmm, 0);
>> VMA_ITERATOR(vmi, mm, 0);
>>
>> uprobe_start_dup_mmap();
>> @@ -678,17 +677,39 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>> goto out;
>> khugepaged_fork(mm, oldmm);
>>
>> - retval = vma_iter_bulk_alloc(&vmi, oldmm->map_count);
>> - if (retval)
>> + /* Use __mt_dup() to efficiently build an identical maple tree. */
>> + retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_NOWAIT | __GFP_NOWARN);
>
> Apparently the flags should be GFP_KERNEL here so that compaction can
> run.
OK, I'll change it to GFP_KERNEL.
>
>> + if (unlikely(retval))
>> goto out;
>>
>> mt_clear_in_rcu(vmi.mas.tree);
>> - for_each_vma(old_vmi, mpnt) {
>> + for_each_vma(vmi, mpnt) {
>> struct file *file;
>>
>> vma_start_write(mpnt);
>> if (mpnt->vm_flags & VM_DONTCOPY) {
>> vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt));
>> +
>> + /*
>> + * Since the new tree is exactly the same as the old one,
>> + * we need to remove the unneeded VMAs.
>> + */
>> + mas_store(&vmi.mas, NULL);
>> +
>> + /*
>> + * Even removing an entry may require memory allocation,
>> + * and if removal fails, we use XA_ZERO_ENTRY to mark
>> + * from which VMA it failed. The case of encountering
>> + * XA_ZERO_ENTRY will be handled in exit_mmap().
>> + */
>> + if (unlikely(mas_is_err(&vmi.mas))) {
>> + retval = xa_err(vmi.mas.node);
>> + mas_reset(&vmi.mas);
>> + if (mas_find(&vmi.mas, ULONG_MAX))
>> + mas_store(&vmi.mas, XA_ZERO_ENTRY);
>> + goto loop_out;
>> + }
>> +
>
> Storing NULL may need extra space as you noted, so we need to be careful
> what happens if we don't have that space. We should have a testcase to
> test this scenario.
>
> mas_store_gfp() should be used with GFP_KERNEL. The VMAs use GFP_KERNEL
> in this function, see vm_area_dup().
>
> Don't use the exit_mmap() path to undo a failed fork. You've added
> checks and complications to the exit path for all tasks in the very
> unlikely event that we run out of memory when we hit a very unlikely
> VM_DONTCOPY flag.
>
> I see the issue with having a portion of the tree with new VMAs that are
> accounted and a portion of the tree that has old VMAs that should not be
> looked at. It was clever to use the XA_ZERO_ENTRY as a stop point, but
> we cannot add that complication to the exit path and then there is the
> OOM race to worry about (maybe, I am not sure since this MM isn't
> active yet).
>
> Using what is done in exit_mmap() and do_vmi_align_munmap() as a
> prototype, we can do something like the *untested* code below:
>
> if (unlikely(mas_is_err(&vmi.mas))) {
> unsigned long max = vmi.index;
>
> retval = xa_err(vmi.mas.node);
> mas_set(&vmi.mas, 0);
> tmp = mas_find(&vmi.mas, ULONG_MAX);
> if (tmp) { /* Not the first VMA failed */
> unsigned long nr_accounted = 0;
>
> unmap_region(mm, &vmi.mas, vma, NULL, mpnt, 0, max, max,
> true);
> do {
> if (vma->vm_flags & VM_ACCOUNT)
> nr_accounted += vma_pages(vma);
> remove_vma(vma, true);
> cond_resched();
> vma = mas_find(&vmi.mas, max - 1);
> } while (vma != NULL);
>
> vm_unacct_memory(nr_accounted);
> }
> __mt_destroy(&mm->mm_mt);
> goto loop_out;
> }
>
> Once exit_mmap() is called, the check for OOM (no vma) will catch that
> nothing is left to do.
>
> It might be worth making an inline function to do this to keep the fork
> code clean. We should test this by detecting a specific task name and
> returning a failure at a given interval:
>
> if (!strcmp(current->comm, "fork_test") {
> ...
> }
Thank you for your suggestion, I will do this in the next version.
>
>
>> continue;
>> }
>> charge = 0;
>> @@ -750,8 +771,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>> hugetlb_dup_vma_private(tmp);
>>
>> /* Link the vma into the MT */
>> - if (vma_iter_bulk_store(&vmi, tmp))
>> - goto fail_nomem_vmi_store;
>> + mas_store(&vmi.mas, tmp);
>>
>> mm->map_count++;
>> if (!(tmp->vm_flags & VM_WIPEONFORK))
>> @@ -778,8 +798,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>> uprobe_end_dup_mmap();
>> return retval;
>>
>> -fail_nomem_vmi_store:
>> - unlink_anon_vmas(tmp);
>> fail_nomem_anon_vma_fork:
>> mpol_put(vma_policy(tmp));
>> fail_nomem_policy:
>> diff --git a/mm/mmap.c b/mm/mmap.c
>> index b56a7f0c9f85..dfc6881be81c 100644
>> --- a/mm/mmap.c
>> +++ b/mm/mmap.c
>> @@ -3196,7 +3196,11 @@ void exit_mmap(struct mm_struct *mm)
>> arch_exit_mmap(mm);
>>
>> vma = mas_find(&mas, ULONG_MAX);
>> - if (!vma) {
>> + /*
>> + * If dup_mmap() fails to remove a VMA marked VM_DONTCOPY,
>> + * xa_is_zero(vma) may be true.
>> + */
>> + if (!vma || xa_is_zero(vma)) {
>> /* Can happen if dup_mmap() received an OOM */
>> mmap_read_unlock(mm);
>> return;
>> @@ -3234,7 +3238,13 @@ void exit_mmap(struct mm_struct *mm)
>> remove_vma(vma, true);
>> count++;
>> cond_resched();
>> - } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);
>> + vma = mas_find(&mas, ULONG_MAX);
>> + /*
>> + * If xa_is_zero(vma) is true, it means that subsequent VMAs
>> + * donot need to be removed. Can happen if dup_mmap() fails to
>> + * remove a VMA marked VM_DONTCOPY.
>> + */
>> + } while (vma != NULL && !xa_is_zero(vma));
>>
>> BUG_ON(count != mm->map_count);
>>
>> --
>> 2.20.1
>>
>
Powered by blists - more mailing lists