Message-ID: <YpDCzvLER9AYJJc8@osiris>
Date:   Fri, 27 May 2022 14:23:42 +0200
From:   Heiko Carstens <hca@...ux.ibm.com>
To:     Peter Xu <peterx@...hat.com>
Cc:     linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        Richard Henderson <rth@...ddle.net>,
        David Hildenbrand <david@...hat.com>,
        Matt Turner <mattst88@...il.com>,
        Albert Ou <aou@...s.berkeley.edu>,
        Michal Simek <monstr@...str.eu>,
        Russell King <linux@...linux.org.uk>,
        Ivan Kokshaysky <ink@...assic.park.msu.ru>,
        linux-riscv@...ts.infradead.org,
        Alexander Gordeev <agordeev@...ux.ibm.com>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        Jonas Bonn <jonas@...thpole.se>, Will Deacon <will@...nel.org>,
        "James E . J . Bottomley" <James.Bottomley@...senpartnership.com>,
        "H . Peter Anvin" <hpa@...or.com>,
        Andrea Arcangeli <aarcange@...hat.com>,
        openrisc@...ts.librecores.org, linux-s390@...r.kernel.org,
        Ingo Molnar <mingo@...hat.com>,
        linux-m68k@...ts.linux-m68k.org,
        Palmer Dabbelt <palmer@...belt.com>,
        Chris Zankel <chris@...kel.net>,
        Peter Zijlstra <peterz@...radead.org>,
        Alistair Popple <apopple@...dia.com>,
        linux-csky@...r.kernel.org, linux-hexagon@...r.kernel.org,
        Vlastimil Babka <vbabka@...e.cz>,
        Thomas Gleixner <tglx@...utronix.de>,
        sparclinux@...r.kernel.org,
        Christian Borntraeger <borntraeger@...ux.ibm.com>,
        Stafford Horne <shorne@...il.com>,
        Michael Ellerman <mpe@...erman.id.au>, x86@...nel.org,
        Thomas Bogendoerfer <tsbogend@...ha.franken.de>,
        Paul Mackerras <paulus@...ba.org>,
        linux-arm-kernel@...ts.infradead.org,
        Sven Schnelle <svens@...ux.ibm.com>,
        Benjamin Herrenschmidt <benh@...nel.crashing.org>,
        linux-xtensa@...ux-xtensa.org, Nicholas Piggin <npiggin@...il.com>,
        linux-sh@...r.kernel.org, Vasily Gorbik <gor@...ux.ibm.com>,
        Borislav Petkov <bp@...en8.de>, linux-mips@...r.kernel.org,
        Max Filippov <jcmvbkbc@...il.com>,
        Helge Deller <deller@....de>, Vineet Gupta <vgupta@...nel.org>,
        Al Viro <viro@...iv.linux.org.uk>,
        Paul Walmsley <paul.walmsley@...ive.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Anton Ivanov <anton.ivanov@...bridgegreys.com>,
        Catalin Marinas <catalin.marinas@....com>,
        linux-um@...ts.infradead.org, linux-alpha@...r.kernel.org,
        Johannes Berg <johannes@...solutions.net>,
        linux-ia64@...r.kernel.org,
        Geert Uytterhoeven <geert@...ux-m68k.org>,
        Dinh Nguyen <dinguyen@...nel.org>, Guo Ren <guoren@...nel.org>,
        linux-snps-arc@...ts.infradead.org,
        Hugh Dickins <hughd@...gle.com>, Rich Felker <dalias@...c.org>,
        Andy Lutomirski <luto@...nel.org>,
        Richard Weinberger <richard@....at>,
        linuxppc-dev@...ts.ozlabs.org, Brian Cain <bcain@...cinc.com>,
        Yoshinori Sato <ysato@...rs.sourceforge.jp>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Stefan Kristiansson <stefan.kristiansson@...nalahti.fi>,
        linux-parisc@...r.kernel.org,
        "David S . Miller" <davem@...emloft.net>,
        Janosch Frank <frankja@...ux.ibm.com>
Subject: Re: [PATCH v3] mm: Avoid unnecessary page fault retires on shared
 memory types

On Tue, May 24, 2022 at 07:45:31PM -0400, Peter Xu wrote:
> I observed that for each of the shared file-backed page faults, we're very
> likely to retry one more time for the 1st write fault upon no page.  It's
> because we'll need to release the mmap lock for dirty rate limit purpose
> with balance_dirty_pages_ratelimited() (in fault_dirty_shared_page()).
> 
> Then after that throttling we return VM_FAULT_RETRY.
> 
> We did that probably because VM_FAULT_RETRY is the only way we can return
> to the fault handler at that time telling it we've released the mmap lock.
> 
> However that's not ideal because it's very likely the fault does not need
> to be retried at all since the pgtable was well installed before the
> throttling, so the next continuous fault (including taking mmap read lock,
> walk the pgtable, etc.) could be in most cases unnecessary.
> 
> It's not only slowing down page faults for shared file-backed, but also add
> more mmap lock contention which is in most cases not needed at all.
> 
> To observe this, one could try to write to some shmem page and look at
> "pgfault" value in /proc/vmstat, then we should expect 2 counts for each
> shmem write simply because we retried, and vm event "pgfault" will capture
> that.
> 
> To make it more efficient, add a new VM_FAULT_COMPLETED return code just to
> show that we've completed the whole fault and released the lock.  It's also
> a hint that we should very possibly not need another fault immediately on
> this page because we've just completed it.
> 
> This patch provides a ~12% perf boost on my aarch64 test VM with a simple
> program sequentially dirtying 400MB shmem file being mmap()ed and these are
> the time it needs:
> 
>   Before: 650.980 ms (+-1.94%)
>   After:  569.396 ms (+-1.38%)
> 
> I believe it could help more than that.
> 
> We need some special care on GUP and the s390 pgfault handler (for gmap
> code before returning from pgfault), the rest changes in the page fault
> handlers should be relatively straightforward.
> 
> Another thing to mention is that mm_account_fault() does take this new
> fault as a generic fault to be accounted, unlike VM_FAULT_RETRY.
> 
> I explicitly didn't touch hmm_vma_fault() and break_ksm() because they do
> not handle VM_FAULT_RETRY even with existing code, so I'm literally keeping
> them as-is.
> 
> Signed-off-by: Peter Xu <peterx@...hat.com>
...
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index e173b6187ad5..9503a7cfaf03 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -339,6 +339,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
>  	unsigned long address;
>  	unsigned int flags;
>  	vm_fault_t fault;
> +	bool need_unlock = true;
>  	bool is_write;
>  
>  	tsk = current;
> @@ -433,6 +434,13 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
>  			goto out_up;
>  		goto out;
>  	}
> +
> +	/* The fault is fully completed (including releasing mmap lock) */
> +	if (fault & VM_FAULT_COMPLETED) {
> +		need_unlock = false;
> +		goto out_gmap;
> +	}
> +
>  	if (unlikely(fault & VM_FAULT_ERROR))
>  		goto out_up;
>  
> @@ -452,6 +460,7 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
>  		mmap_read_lock(mm);
>  		goto retry;
>  	}
> +out_gmap:
>  	if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
>  		address =  __gmap_link(gmap, current->thread.gmap_addr,
>  				       address);
> @@ -466,7 +475,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access)
>  	}
>  	fault = 0;
>  out_up:
> -	mmap_read_unlock(mm);
> +	if (need_unlock)
> +		mmap_read_unlock(mm);
>  out:

This seems to be incorrect. __gmap_link() requires the mmap_lock to be
held. Christian, Janosch, or David, could you please check?

Powered by blists - more mailing lists