Message-ID: <5B585D9E.307@huawei.com>
Date:   Wed, 25 Jul 2018 19:23:10 +0800
From:   zhong jiang <zhongjiang@...wei.com>
To:     Laurent Dufour <ldufour@...ux.vnet.ibm.com>
CC:     <akpm@...ux-foundation.org>, <mhocko@...nel.org>,
        <peterz@...radead.org>, <kirill@...temov.name>,
        <ak@...ux.intel.com>, <dave@...olabs.net>, <jack@...e.cz>,
        Matthew Wilcox <willy@...radead.org>,
        <khandual@...ux.vnet.ibm.com>, <aneesh.kumar@...ux.vnet.ibm.com>,
        <benh@...nel.crashing.org>, <mpe@...erman.id.au>,
        <paulus@...ba.org>, "Thomas Gleixner" <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, <hpa@...or.com>,
        Will Deacon <will.deacon@....com>,
        Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
        <sergey.senozhatsky.work@...il.com>,
        "Andrea Arcangeli" <aarcange@...hat.com>,
        Alexei Starovoitov <alexei.starovoitov@...il.com>,
        <kemi.wang@...el.com>, Daniel Jordan <daniel.m.jordan@...cle.com>,
        David Rientjes <rientjes@...gle.com>,
        "Jerome Glisse" <jglisse@...hat.com>,
        Ganesh Mahendran <opensource.ganesh@...il.com>,
        Minchan Kim <minchan@...nel.org>,
        Punit Agrawal <punitagrawal@...il.com>,
        vinayak menon <vinayakm.list@...il.com>,
        Yang Shi <yang.shi@...ux.alibaba.com>,
        <linux-kernel@...r.kernel.org>, <linux-mm@...ck.org>,
        <haren@...ux.vnet.ibm.com>, <npiggin@...il.com>,
        <bsingharora@...il.com>, <paulmck@...ux.vnet.ibm.com>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        <linuxppc-dev@...ts.ozlabs.org>, <x86@...nel.org>
Subject: Re: [PATCH v11 19/26] mm: provide speculative fault infrastructure

On 2018/7/25 18:44, Laurent Dufour wrote:
>
> On 25/07/2018 11:04, zhong jiang wrote:
>> On 2018/7/25 0:10, Laurent Dufour wrote:
>>> On 24/07/2018 16:26, zhong jiang wrote:
>>>> On 2018/5/17 19:06, Laurent Dufour wrote:
>>>>> From: Peter Zijlstra <peterz@...radead.org>
>>>>>
>>>>> Provide infrastructure to do a speculative fault (not holding
>>>>> mmap_sem).
>>>>>
>>>>> The not holding of mmap_sem means we can race against VMA
>>>>> change/removal and page-table destruction. We use the SRCU VMA freeing
>>>>> to keep the VMA around. We use the VMA seqcount to detect change
>>>>> (including umapping / page-table deletion) and we use gup_fast() style
>>>>> page-table walking to deal with page-table races.
>>>>>
>>>>> Once we've obtained the page and are ready to update the PTE, we
>>>>> validate if the state we started the fault with is still valid, if
>>>>> not, we'll fail the fault with VM_FAULT_RETRY, otherwise we update the
>>>>> PTE and we're done.
>>>>>
>>>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
>>>>>
>>>>> [Manage the newly introduced pte_spinlock() for speculative page
>>>>>  fault to fail if the VMA is touched in our back]
>>>>> [Rename vma_is_dead() to vma_has_changed() and declare it here]
>>>>> [Fetch p4d and pud]
>>>>> [Set vmd.sequence in __handle_mm_fault()]
>>>>> [Abort speculative path when handle_userfault() has to be called]
>>>>> [Add additional VMA's flags checks in handle_speculative_fault()]
>>>>> [Clear FAULT_FLAG_ALLOW_RETRY in handle_speculative_fault()]
>>>>> [Don't set vmf->pte and vmf->ptl if pte_map_lock() failed]
>>>>> [Remove warning comment about waiting for !seq&1 since we don't want
>>>>>  to wait]
>>>>> [Remove warning about no huge page support, mention it explictly]
>>>>> [Don't call do_fault() in the speculative path as __do_fault() calls
>>>>>  vma->vm_ops->fault() which may want to release mmap_sem]
>>>>> [Only vm_fault pointer argument for vma_has_changed()]
>>>>> [Fix check against huge page, calling pmd_trans_huge()]
>>>>> [Use READ_ONCE() when reading VMA's fields in the speculative path]
>>>>> [Explicitly check for __HAVE_ARCH_PTE_SPECIAL as we can't support for
>>>>>  processing done in vm_normal_page()]
>>>>> [Check that vma->anon_vma is already set when starting the speculative
>>>>>  path]
>>>>> [Check for memory policy as we can't support MPOL_INTERLEAVE case due to
>>>>>  the processing done in mpol_misplaced()]
>>>>> [Don't support VMA growing up or down]
>>>>> [Move check on vm_sequence just before calling handle_pte_fault()]
>>>>> [Don't build SPF services if !CONFIG_SPECULATIVE_PAGE_FAULT]
>>>>> [Add mem cgroup oom check]
>>>>> [Use READ_ONCE to access p*d entries]
>>>>> [Replace deprecated ACCESS_ONCE() by READ_ONCE() in vma_has_changed()]
>>>>> [Don't fetch pte again in handle_pte_fault() when running the speculative
>>>>>  path]
>>>>> [Check PMD against concurrent collapsing operation]
>>>>> [Try spin lock the pte during the speculative path to avoid deadlock with
>>>>>  other CPU's invalidating the TLB and requiring this CPU to catch the
>>>>>  inter processor's interrupt]
>>>>> [Move define of FAULT_FLAG_SPECULATIVE here]
>>>>> [Introduce __handle_speculative_fault() and add a check against
>>>>>  mm->mm_users in handle_speculative_fault() defined in mm.h]
>>>>> Signed-off-by: Laurent Dufour <ldufour@...ux.vnet.ibm.com>
>>>>> ---
>>>>>  include/linux/hugetlb_inline.h |   2 +-
>>>>>  include/linux/mm.h             |  30 ++++
>>>>>  include/linux/pagemap.h        |   4 +-
>>>>>  mm/internal.h                  |  16 +-
>>>>>  mm/memory.c                    | 340 ++++++++++++++++++++++++++++++++++++++++-
>>>>>  5 files changed, 385 insertions(+), 7 deletions(-)
>>>>>
>>>>> diff --git a/include/linux/hugetlb_inline.h b/include/linux/hugetlb_inline.h
>>>>> index 0660a03d37d9..9e25283d6fc9 100644
>>>>> --- a/include/linux/hugetlb_inline.h
>>>>> +++ b/include/linux/hugetlb_inline.h
>>>>> @@ -8,7 +8,7 @@
>>>>>  
>>>>>  static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma)
>>>>>  {
>>>>> -	return !!(vma->vm_flags & VM_HUGETLB);
>>>>> +	return !!(READ_ONCE(vma->vm_flags) & VM_HUGETLB);
>>>>>  }
>>>>>  
>>>>>  #else
>>>>> diff --git a/include/linux/mm.h b/include/linux/mm.h
>>>>> index 05cbba70104b..31acf98a7d92 100644
>>>>> --- a/include/linux/mm.h
>>>>> +++ b/include/linux/mm.h
>>>>> @@ -315,6 +315,7 @@ extern pgprot_t protection_map[16];
>>>>>  #define FAULT_FLAG_USER		0x40	/* The fault originated in userspace */
>>>>>  #define FAULT_FLAG_REMOTE	0x80	/* faulting for non current tsk/mm */
>>>>>  #define FAULT_FLAG_INSTRUCTION  0x100	/* The fault was during an instruction fetch */
>>>>> +#define FAULT_FLAG_SPECULATIVE	0x200	/* Speculative fault, not holding mmap_sem */
>>>>>  
>>>>>  #define FAULT_FLAG_TRACE \
>>>>>  	{ FAULT_FLAG_WRITE,		"WRITE" }, \
>>>>> @@ -343,6 +344,10 @@ struct vm_fault {
>>>>>  	gfp_t gfp_mask;			/* gfp mask to be used for allocations */
>>>>>  	pgoff_t pgoff;			/* Logical page offset based on vma */
>>>>>  	unsigned long address;		/* Faulting virtual address */
>>>>> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
>>>>> +	unsigned int sequence;
>>>>> +	pmd_t orig_pmd;			/* value of PMD at the time of fault */
>>>>> +#endif
>>>>>  	pmd_t *pmd;			/* Pointer to pmd entry matching
>>>>>  					 * the 'address' */
>>>>>  	pud_t *pud;			/* Pointer to pud entry matching
>>>>> @@ -1415,6 +1420,31 @@ int invalidate_inode_page(struct page *page);
>>>>>  #ifdef CONFIG_MMU
>>>>>  extern int handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
>>>>>  		unsigned int flags);
>>>>> +
>>>>> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
>>>>> +extern int __handle_speculative_fault(struct mm_struct *mm,
>>>>> +				      unsigned long address,
>>>>> +				      unsigned int flags);
>>>>> +static inline int handle_speculative_fault(struct mm_struct *mm,
>>>>> +					   unsigned long address,
>>>>> +					   unsigned int flags)
>>>>> +{
>>>>> +	/*
>>>>> +	 * Try speculative page fault for multithreaded user space task only.
>>>>> +	 */
>>>>> +	if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1)
>>>>> +		return VM_FAULT_RETRY;
>>>>> +	return __handle_speculative_fault(mm, address, flags);
>>>>> +}
>>>>> +#else
>>>>> +static inline int handle_speculative_fault(struct mm_struct *mm,
>>>>> +					   unsigned long address,
>>>>> +					   unsigned int flags)
>>>>> +{
>>>>> +	return VM_FAULT_RETRY;
>>>>> +}
>>>>> +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */
>>>>> +
>>>>>  extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm,
>>>>>  			    unsigned long address, unsigned int fault_flags,
>>>>>  			    bool *unlocked);
>>>>> diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
>>>>> index b1bd2186e6d2..6e2aa4e79af7 100644
>>>>> --- a/include/linux/pagemap.h
>>>>> +++ b/include/linux/pagemap.h
>>>>> @@ -456,8 +456,8 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma,
>>>>>  	pgoff_t pgoff;
>>>>>  	if (unlikely(is_vm_hugetlb_page(vma)))
>>>>>  		return linear_hugepage_index(vma, address);
>>>>> -	pgoff = (address - vma->vm_start) >> PAGE_SHIFT;
>>>>> -	pgoff += vma->vm_pgoff;
>>>>> +	pgoff = (address - READ_ONCE(vma->vm_start)) >> PAGE_SHIFT;
>>>>> +	pgoff += READ_ONCE(vma->vm_pgoff);
>>>>>  	return pgoff;
>>>>>  }
>>>>>  
>>>>> diff --git a/mm/internal.h b/mm/internal.h
>>>>> index fb2667b20f0a..10b188c87fa4 100644
>>>>> --- a/mm/internal.h
>>>>> +++ b/mm/internal.h
>>>>> @@ -44,7 +44,21 @@ int do_swap_page(struct vm_fault *vmf);
>>>>>  extern struct vm_area_struct *get_vma(struct mm_struct *mm,
>>>>>  				      unsigned long addr);
>>>>>  extern void put_vma(struct vm_area_struct *vma);
>>>>> -#endif
>>>>> +
>>>>> +static inline bool vma_has_changed(struct vm_fault *vmf)
>>>>> +{
>>>>> +	int ret = RB_EMPTY_NODE(&vmf->vma->vm_rb);
>>>>> +	unsigned int seq = READ_ONCE(vmf->vma->vm_sequence.sequence);
>>>>> +
>>>>> +	/*
>>>>> +	 * Matches both the wmb in write_seqlock_{begin,end}() and
>>>>> +	 * the wmb in vma_rb_erase().
>>>>> +	 */
>>>>> +	smp_rmb();
>>>>> +
>>>>> +	return ret || seq != vmf->sequence;
>>>>> +}
>>>>> +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */
>>>>>  
>>>>>  void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
>>>>>  		unsigned long floor, unsigned long ceiling);
>>>>> diff --git a/mm/memory.c b/mm/memory.c
>>>>> index ab32b0b4bd69..7bbbb8c7b9cd 100644
>>>>> --- a/mm/memory.c
>>>>> +++ b/mm/memory.c
>>>>> @@ -769,7 +769,8 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
>>>>>  	if (page)
>>>>>  		dump_page(page, "bad pte");
>>>>>  	pr_alert("addr:%p vm_flags:%08lx anon_vma:%p mapping:%p index:%lx\n",
>>>>> -		 (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index);
>>>>> +		 (void *)addr, READ_ONCE(vma->vm_flags), vma->anon_vma,
>>>>> +		 mapping, index);
>>>>>  	pr_alert("file:%pD fault:%pf mmap:%pf readpage:%pf\n",
>>>>>  		 vma->vm_file,
>>>>>  		 vma->vm_ops ? vma->vm_ops->fault : NULL,
>>>>> @@ -2306,6 +2307,118 @@ int apply_to_page_range(struct mm_struct *mm, unsigned long addr,
>>>>>  }
>>>>>  EXPORT_SYMBOL_GPL(apply_to_page_range);
>>>>>  
>>>>> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
>>>>> +static bool pte_spinlock(struct vm_fault *vmf)
>>>>> +{
>>>>> +	bool ret = false;
>>>>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>>>>> +	pmd_t pmdval;
>>>>> +#endif
>>>>> +
>>>>> +	/* Check if vma is still valid */
>>>>> +	if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
>>>>> +		vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
>>>>> +		spin_lock(vmf->ptl);
>>>>> +		return true;
>>>>> +	}
>>>>> +
>>>>> +again:
>>>>> +	local_irq_disable();
>>>>> +	if (vma_has_changed(vmf))
>>>>> +		goto out;
>>>>> +
>>>>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>>>>> +	/*
>>>>> +	 * We check if the pmd value is still the same to ensure that there
>>>>> +	 * is not a huge collapse operation in progress in our back.
>>>>> +	 */
>>>>> +	pmdval = READ_ONCE(*vmf->pmd);
>>>>> +	if (!pmd_same(pmdval, vmf->orig_pmd))
>>>>> +		goto out;
>>>>> +#endif
>>>>> +
>>>>> +	vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
>>>>> +	if (unlikely(!spin_trylock(vmf->ptl))) {
>>>>> +		local_irq_enable();
>>>>> +		goto again;
>>>>> +	}
>>>>> +
>>>>> +	if (vma_has_changed(vmf)) {
>>>>> +		spin_unlock(vmf->ptl);
>>>>> +		goto out;
>>>>> +	}
>>>>> +
>>>>> +	ret = true;
>>>>> +out:
>>>>> +	local_irq_enable();
>>>>> +	return ret;
>>>>> +}
>>>>> +
>>>>> +static bool pte_map_lock(struct vm_fault *vmf)
>>>>> +{
>>>>> +	bool ret = false;
>>>>> +	pte_t *pte;
>>>>> +	spinlock_t *ptl;
>>>>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>>>>> +	pmd_t pmdval;
>>>>> +#endif
>>>>> +
>>>>> +	if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
>>>>> +		vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
>>>>> +					       vmf->address, &vmf->ptl);
>>>>> +		return true;
>>>>> +	}
>>>>> +
>>>>> +	/*
>>>>> +	 * The first vma_has_changed() guarantees the page-tables are still
>>>>> +	 * valid, having IRQs disabled ensures they stay around, hence the
>>>>> +	 * second vma_has_changed() to make sure they are still valid once
>>>>> +	 * we've got the lock. After that a concurrent zap_pte_range() will
>>>>> +	 * block on the PTL and thus we're safe.
>>>>> +	 */
>>>>> +again:
>>>>> +	local_irq_disable();
>>>>> +	if (vma_has_changed(vmf))
>>>>> +		goto out;
>>>>> +
>>>>> +#ifdef CONFIG_TRANSPARENT_HUGEPAGE
>>>>> +	/*
>>>>> +	 * We check if the pmd value is still the same to ensure that there
>>>>> +	 * is not a huge collapse operation in progress in our back.
>>>>> +	 */
>>>>> +	pmdval = READ_ONCE(*vmf->pmd);
>>>>> +	if (!pmd_same(pmdval, vmf->orig_pmd))
>>>>> +		goto out;
>>>>> +#endif
>>>>> +
>>>>> +	/*
>>>>> +	 * Same as pte_offset_map_lock() except that we call
>>>>> +	 * spin_trylock() in place of spin_lock() to avoid race with
>>>>> +	 * unmap path which may have the lock and wait for this CPU
>>>>> +	 * to invalidate TLB but this CPU has irq disabled.
>>>>> +	 * Since we are in a speculative patch, accept it could fail
>>>>> +	 */
>>>>> +	ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
>>>>> +	pte = pte_offset_map(vmf->pmd, vmf->address);
>>>>> +	if (unlikely(!spin_trylock(ptl))) {
>>>>> +		pte_unmap(pte);
>>>>> +		local_irq_enable();
>>>>> +		goto again;
>>>>> +	}
>>>>> +
>>>>> +	if (vma_has_changed(vmf)) {
>>>>> +		pte_unmap_unlock(pte, ptl);
>>>>> +		goto out;
>>>>> +	}
>>>>> +
>>>>> +	vmf->pte = pte;
>>>>> +	vmf->ptl = ptl;
>>>>> +	ret = true;
>>>>> +out:
>>>>> +	local_irq_enable();
>>>>> +	return ret;
>>>>> +}
>>>>> +#else
>>>>>  static inline bool pte_spinlock(struct vm_fault *vmf)
>>>>>  {
>>>>>  	vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
>>>>> @@ -2319,6 +2432,7 @@ static inline bool pte_map_lock(struct vm_fault *vmf)
>>>>>  				       vmf->address, &vmf->ptl);
>>>>>  	return true;
>>>>>  }
>>>>> +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */
>>>>>  
>>>>>  /*
>>>>>   * handle_pte_fault chooses page fault handler according to an entry which was
>>>>> @@ -3208,6 +3322,14 @@ static int do_anonymous_page(struct vm_fault *vmf)
>>>>>  		ret = check_stable_address_space(vma->vm_mm);
>>>>>  		if (ret)
>>>>>  			goto unlock;
>>>>> +		/*
>>>>> +		 * Don't call the userfaultfd during the speculative path.
>>>>> +		 * We already checked for the VMA to not be managed through
>>>>> +		 * userfaultfd, but it may be set in our back once we have lock
>>>>> +		 * the pte. In such a case we can ignore it this time.
>>>>> +		 */
>>>>> +		if (vmf->flags & FAULT_FLAG_SPECULATIVE)
>>>>> +			goto setpte;
>>>>>  		/* Deliver the page fault to userland, check inside PT lock */
>>>>>  		if (userfaultfd_missing(vma)) {
>>>>>  			pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>>> @@ -3249,7 +3371,7 @@ static int do_anonymous_page(struct vm_fault *vmf)
>>>>>  		goto unlock_and_release;
>>>>>  
>>>>>  	/* Deliver the page fault to userland, check inside PT lock */
>>>>> -	if (userfaultfd_missing(vma)) {
>>>>> +	if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && userfaultfd_missing(vma)) {
>>>>>  		pte_unmap_unlock(vmf->pte, vmf->ptl);
>>>>>  		mem_cgroup_cancel_charge(page, memcg, false);
>>>>>  		put_page(page);
>>>>> @@ -3994,13 +4116,22 @@ static int handle_pte_fault(struct vm_fault *vmf)
>>>>>  
>>>>>  	if (unlikely(pmd_none(*vmf->pmd))) {
>>>>>  		/*
>>>>> +		 * In the case of the speculative page fault handler we abort
>>>>> +		 * the speculative path immediately as the pmd is probably
>>>>> +		 * in the way to be converted in a huge one. We will try
>>>>> +		 * again holding the mmap_sem (which implies that the collapse
>>>>> +		 * operation is done).
>>>>> +		 */
>>>>> +		if (vmf->flags & FAULT_FLAG_SPECULATIVE)
>>>>> +			return VM_FAULT_RETRY;
>>>>> +		/*
>>>>>  		 * Leave __pte_alloc() until later: because vm_ops->fault may
>>>>>  		 * want to allocate huge page, and if we expose page table
>>>>>  		 * for an instant, it will be difficult to retract from
>>>>>  		 * concurrent faults and from rmap lookups.
>>>>>  		 */
>>>>>  		vmf->pte = NULL;
>>>>> -	} else {
>>>>> +	} else if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) {
>>>>>  		/* See comment in pte_alloc_one_map() */
>>>>>  		if (pmd_devmap_trans_unstable(vmf->pmd))
>>>>>  			return 0;
>>>>> @@ -4009,6 +4140,9 @@ static int handle_pte_fault(struct vm_fault *vmf)
>>>>>  		 * pmd from under us anymore at this point because we hold the
>>>>>  		 * mmap_sem read mode and khugepaged takes it in write mode.
>>>>>  		 * So now it's safe to run pte_offset_map().
>>>>> +		 * This is not applicable to the speculative page fault handler
>>>>> +		 * but in that case, the pte is fetched earlier in
>>>>> +		 * handle_speculative_fault().
>>>>>  		 */
>>>>>  		vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
>>>>>  		vmf->orig_pte = *vmf->pte;
>>>>> @@ -4031,6 +4165,8 @@ static int handle_pte_fault(struct vm_fault *vmf)
>>>>>  	if (!vmf->pte) {
>>>>>  		if (vma_is_anonymous(vmf->vma))
>>>>>  			return do_anonymous_page(vmf);
>>>>> +		else if (vmf->flags & FAULT_FLAG_SPECULATIVE)
>>>>> +			return VM_FAULT_RETRY;
>>>>>  		else
>>>>>  			return do_fault(vmf);
>>>>>  	}
>>>>> @@ -4128,6 +4264,9 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
>>>>>  	vmf.pmd = pmd_alloc(mm, vmf.pud, address);
>>>>>  	if (!vmf.pmd)
>>>>>  		return VM_FAULT_OOM;
>>>>> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
>>>>> +	vmf.sequence = raw_read_seqcount(&vma->vm_sequence);
>>>>> +#endif
>>>>>  	if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
>>>>>  		ret = create_huge_pmd(&vmf);
>>>>>  		if (!(ret & VM_FAULT_FALLBACK))
>>>>> @@ -4161,6 +4300,201 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
>>>>>  	return handle_pte_fault(&vmf);
>>>>>  }
>>>>>  
>>>>> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
>>>>> +/*
>>>>> + * Tries to handle the page fault in a speculative way, without grabbing the
>>>>> + * mmap_sem.
>>>>> + */
>>>>> +int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,
>>>>> +			       unsigned int flags)
>>>>> +{
>>>>> +	struct vm_fault vmf = {
>>>>> +		.address = address,
>>>>> +	};
>>>>> +	pgd_t *pgd, pgdval;
>>>>> +	p4d_t *p4d, p4dval;
>>>>> +	pud_t pudval;
>>>>> +	int seq, ret = VM_FAULT_RETRY;
>>>>> +	struct vm_area_struct *vma;
>>>>> +#ifdef CONFIG_NUMA
>>>>> +	struct mempolicy *pol;
>>>>> +#endif
>>>>> +
>>>>> +	/* Clear flags that may lead to release the mmap_sem to retry */
>>>>> +	flags &= ~(FAULT_FLAG_ALLOW_RETRY|FAULT_FLAG_KILLABLE);
>>>>> +	flags |= FAULT_FLAG_SPECULATIVE;
>>>>> +
>>>>> +	vma = get_vma(mm, address);
>>>>> +	if (!vma)
>>>>> +		return ret;
>>>>> +
>>>>> +	seq = raw_read_seqcount(&vma->vm_sequence); /* rmb <-> seqlock,vma_rb_erase() */
>>>>> +	if (seq & 1)
>>>>> +		goto out_put;
>>>>> +
>>>>> +	/*
>>>>> +	 * Can't call vm_ops service has we don't know what they would do
>>>>> +	 * with the VMA.
>>>>> +	 * This include huge page from hugetlbfs.
>>>>> +	 */
>>>>> +	if (vma->vm_ops)
>>>>> +		goto out_put;
>>>>> +
>>>>   Hi   Laurent
>>>>    
>>>>    I think that most of pagefault will leave here.   Is there any case  need to skip ?
>>>>   I have tested the following  patch, it work well.
>>> Hi Zhong,
>>>
>>> Well this will allow file mapping to be handle in a speculative way, but that's
>>> a bit dangerous today as there is no guaranty that the vm_ops.vm_fault()
>>> operation will be fair.
>>>
>>> In the case of the anonymous file mapping that's often not a problem, depending
>>> on the underlying file system, but there are so many cases to check and this is
>>> hard to say this can be done in a speculative way as is.
>>  This patch say that spf just handle anonyous page. but I find that do_swap_page
>>  also maybe release the mmap_sem without FAULT_FLAG_RETRY_NOWAIT. why is it safe
>>  to handle the case. I think that the case is similar to file page. Maybe I miss
>>  something else.
> do_swap_page() may released the mmap_sem through the call to
> __lock_page_or_retry(), but this can only happen if FAULT_FLAG_ALLOW_RETRY or
> FAULT_FLAG_KILLABLE are set and they are unset in __handle_speculative_fault().
 For spf. Indeed. Thank you for clarification.
>>  I test the patches and find just only 18% of the pagefault will enter into the
>>  speculative page fault during a process startup. As I had said. most of pagefault
>>  will be handled by ops->fault. I do not know the data you had posted is how to get.
> I do agree that handling file mapping will be required, but this will add more
> complexity to this series, since we need a way for drivers to tell they are
> compatible with the speculative path.
 As the above mentioned. the specualtive page fault do not pass FAULT_FLAG_ALLOW_RETRY.
 In other words, File page will not refer to release mmap_sem for spf.
 
 but I am still not quite clear that what should drivers do to compatible with speculatve path.
 The speculative path should not refer to the mmap_sem for filemap_fault.

 Thanks,
 zhong jiang
> May be I should give it a try on the next send.
 Ok, I will try.
> For my information, what was the performance improvement you seen when handling
> file page faulting this way ?
 I am sorry that. It is the data that Ganesh test the launch time on Andriod.
> Thanks,
> Laurent.
>
>>
>>  Thanks
>>  zhong jiang
>>> The huge work to do is to double check that all the code called by
>>> vm_ops.fault() is not dealing with the mmap_sem, which could be handled using
>>> FAULT_FLAG_RETRY_NOWAIT, and care is also needed about the resources that code
>>> is managing as it may assume that it is under the protection of the mmap_sem in
>>> read mode, and that can be done implicitly.
>>>
>>> Cheers,
>>> Laurent.
>>>
>>>> diff --git a/mm/memory.c b/mm/memory.c
>>>> index 936128b..9bc1545 100644
>>>>  @@ -3893,8 +3898,6 @@ static int handle_pte_fault(struct fault_env *fe)
>>>>         if (!fe->pte) {
>>>>                 if (vma_is_anonymous(fe->vma))
>>>>                         return do_anonymous_page(fe);
>>>> -               else if (fe->flags & FAULT_FLAG_SPECULATIVE)
>>>> -                       return VM_FAULT_RETRY;
>>>>                 else
>>>>                         return do_fault(fe);
>>>>         }
>>>> @@ -4026,20 +4029,11 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,
>>>>                 goto out_put;
>>>>         }
>>>>         /*
>>>> -        * Can't call vm_ops service has we don't know what they would do
>>>> -        * with the VMA.
>>>> -        * This include huge page from hugetlbfs.
>>>> -        */
>>>> -       if (vma->vm_ops) {
>>>> -               trace_spf_vma_notsup(_RET_IP_, vma, address);
>>>> -               goto out_put;
>>>> -       }
>>>>
>>>>
>>>> Thanks
>>>> zhong jiang
>>>>> +	/*
>>>>> +	 * __anon_vma_prepare() requires the mmap_sem to be held
>>>>> +	 * because vm_next and vm_prev must be safe. This can't be guaranteed
>>>>> +	 * in the speculative path.
>>>>> +	 */
>>>>> +	if (unlikely(!vma->anon_vma))
>>>>> +		goto out_put;
>>>>> +
>>>>> +	vmf.vma_flags = READ_ONCE(vma->vm_flags);
>>>>> +	vmf.vma_page_prot = READ_ONCE(vma->vm_page_prot);
>>>>> +
>>>>> +	/* Can't call userland page fault handler in the speculative path */
>>>>> +	if (unlikely(vmf.vma_flags & VM_UFFD_MISSING))
>>>>> +		goto out_put;
>>>>> +
>>>>> +	if (vmf.vma_flags & VM_GROWSDOWN || vmf.vma_flags & VM_GROWSUP)
>>>>> +		/*
>>>>> +		 * This could be detected by the check address against VMA's
>>>>> +		 * boundaries but we want to trace it as not supported instead
>>>>> +		 * of changed.
>>>>> +		 */
>>>>> +		goto out_put;
>>>>> +
>>>>> +	if (address < READ_ONCE(vma->vm_start)
>>>>> +	    || READ_ONCE(vma->vm_end) <= address)
>>>>> +		goto out_put;
>>>>> +
>>>>> +	if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
>>>>> +				       flags & FAULT_FLAG_INSTRUCTION,
>>>>> +				       flags & FAULT_FLAG_REMOTE)) {
>>>>> +		ret = VM_FAULT_SIGSEGV;
>>>>> +		goto out_put;
>>>>> +	}
>>>>> +
>>>>> +	/* This is one is required to check that the VMA has write access set */
>>>>> +	if (flags & FAULT_FLAG_WRITE) {
>>>>> +		if (unlikely(!(vmf.vma_flags & VM_WRITE))) {
>>>>> +			ret = VM_FAULT_SIGSEGV;
>>>>> +			goto out_put;
>>>>> +		}
>>>>> +	} else if (unlikely(!(vmf.vma_flags & (VM_READ|VM_EXEC|VM_WRITE)))) {
>>>>> +		ret = VM_FAULT_SIGSEGV;
>>>>> +		goto out_put;
>>>>> +	}
>>>>> +
>>>>> +#ifdef CONFIG_NUMA
>>>>> +	/*
>>>>> +	 * MPOL_INTERLEAVE implies additional checks in
>>>>> +	 * mpol_misplaced() which are not compatible with the
>>>>> +	 *speculative page fault processing.
>>>>> +	 */
>>>>> +	pol = __get_vma_policy(vma, address);
>>>>> +	if (!pol)
>>>>> +		pol = get_task_policy(current);
>>>>> +	if (pol && pol->mode == MPOL_INTERLEAVE)
>>>>> +		goto out_put;
>>>>> +#endif
>>>>> +
>>>>> +	/*
>>>>> +	 * Do a speculative lookup of the PTE entry.
>>>>> +	 */
>>>>> +	local_irq_disable();
>>>>> +	pgd = pgd_offset(mm, address);
>>>>> +	pgdval = READ_ONCE(*pgd);
>>>>> +	if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval)))
>>>>> +		goto out_walk;
>>>>> +
>>>>> +	p4d = p4d_offset(pgd, address);
>>>>> +	p4dval = READ_ONCE(*p4d);
>>>>> +	if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval)))
>>>>> +		goto out_walk;
>>>>> +
>>>>> +	vmf.pud = pud_offset(p4d, address);
>>>>> +	pudval = READ_ONCE(*vmf.pud);
>>>>> +	if (pud_none(pudval) || unlikely(pud_bad(pudval)))
>>>>> +		goto out_walk;
>>>>> +
>>>>> +	/* Huge pages at PUD level are not supported. */
>>>>> +	if (unlikely(pud_trans_huge(pudval)))
>>>>> +		goto out_walk;
>>>>> +
>>>>> +	vmf.pmd = pmd_offset(vmf.pud, address);
>>>>> +	vmf.orig_pmd = READ_ONCE(*vmf.pmd);
>>>>> +	/*
>>>>> +	 * pmd_none could mean that a hugepage collapse is in progress
>>>>> +	 * in our back as collapse_huge_page() mark it before
>>>>> +	 * invalidating the pte (which is done once the IPI is catched
>>>>> +	 * by all CPU and we have interrupt disabled).
>>>>> +	 * For this reason we cannot handle THP in a speculative way since we
>>>>> +	 * can't safely indentify an in progress collapse operation done in our
>>>>> +	 * back on that PMD.
>>>>> +	 * Regarding the order of the following checks, see comment in
>>>>> +	 * pmd_devmap_trans_unstable()
>>>>> +	 */
>>>>> +	if (unlikely(pmd_devmap(vmf.orig_pmd) ||
>>>>> +		     pmd_none(vmf.orig_pmd) || pmd_trans_huge(vmf.orig_pmd) ||
>>>>> +		     is_swap_pmd(vmf.orig_pmd)))
>>>>> +		goto out_walk;
>>>>> +
>>>>> +	/*
>>>>> +	 * The above does not allocate/instantiate page-tables because doing so
>>>>> +	 * would lead to the possibility of instantiating page-tables after
>>>>> +	 * free_pgtables() -- and consequently leaking them.
>>>>> +	 *
>>>>> +	 * The result is that we take at least one !speculative fault per PMD
>>>>> +	 * in order to instantiate it.
>>>>> +	 */
>>>>> +
>>>>> +	vmf.pte = pte_offset_map(vmf.pmd, address);
>>>>> +	vmf.orig_pte = READ_ONCE(*vmf.pte);
>>>>> +	barrier(); /* See comment in handle_pte_fault() */
>>>>> +	if (pte_none(vmf.orig_pte)) {
>>>>> +		pte_unmap(vmf.pte);
>>>>> +		vmf.pte = NULL;
>>>>> +	}
>>>>> +
>>>>> +	vmf.vma = vma;
>>>>> +	vmf.pgoff = linear_page_index(vma, address);
>>>>> +	vmf.gfp_mask = __get_fault_gfp_mask(vma);
>>>>> +	vmf.sequence = seq;
>>>>> +	vmf.flags = flags;
>>>>> +
>>>>> +	local_irq_enable();
>>>>> +
>>>>> +	/*
>>>>> +	 * We need to re-validate the VMA after checking the bounds, otherwise
>>>>> +	 * we might have a false positive on the bounds.
>>>>> +	 */
>>>>> +	if (read_seqcount_retry(&vma->vm_sequence, seq))
>>>>> +		goto out_put;
>>>>> +
>>>>> +	mem_cgroup_oom_enable();
>>>>> +	ret = handle_pte_fault(&vmf);
>>>>> +	mem_cgroup_oom_disable();
>>>>> +
>>>>> +	put_vma(vma);
>>>>> +
>>>>> +	/*
>>>>> +	 * The task may have entered a memcg OOM situation but
>>>>> +	 * if the allocation error was handled gracefully (no
>>>>> +	 * VM_FAULT_OOM), there is no need to kill anything.
>>>>> +	 * Just clean up the OOM state peacefully.
>>>>> +	 */
>>>>> +	if (task_in_memcg_oom(current) && !(ret & VM_FAULT_OOM))
>>>>> +		mem_cgroup_oom_synchronize(false);
>>>>> +	return ret;
>>>>> +
>>>>> +out_walk:
>>>>> +	local_irq_enable();
>>>>> +out_put:
>>>>> +	put_vma(vma);
>>>>> +	return ret;
>>>>> +}
>>>>> +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */
>>>>> +
>>>>>  /*
>>>>>   * By the time we get here, we already hold the mm semaphore
>>>>>   *
>>> .
>>>
>>
>
> .
>

Powered by blists - more mailing lists