linux-kernel - Re: [PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they exist

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Tue, 03 Jan 2012 12:45:45 -0500
From:	KOSAKI Motohiro <kosaki.motohiro@...il.com>
To:	Gilad Ben-Yossef <gilad@...yossef.com>
CC:	linux-kernel@...r.kernel.org, Chris Metcalf <cmetcalf@...era.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Russell King <linux@....linux.org.uk>, linux-mm@...ck.org,
	Pekka Enberg <penberg@...nel.org>,
	Matt Mackall <mpm@...enic.com>,
	Sasha Levin <levinsasha928@...il.com>,
	Rik van Riel <riel@...hat.com>,
	Andi Kleen <andi@...stfloor.org>, Mel Gorman <mel@....ul.ie>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	linux-fsdevel@...r.kernel.org, Avi Kivity <avi@...hat.com>
Subject: Re: [PATCH v5 7/8] mm: Only IPI CPUs to drain local pages if they
 exist

(1/2/12 5:24 AM), Gilad Ben-Yossef wrote:
> Calculate a cpumask of CPUs with per-cpu pages in any zone
> and only send an IPI requesting CPUs to drain these pages
> to the buddy allocator if they actually have pages when
> asked to flush.
> 
> This patch saves 99% of IPIs asking to drain per-cpu
> pages in case of severe memory preassure that leads
> to OOM since in these cases multiple, possibly concurrent,
> allocation requests end up in the direct reclaim code
> path so when the per-cpu pages end up reclaimed on first
> allocation failure for most of the proceeding allocation
> attempts until the memory pressure is off (possibly via
> the OOM killer) there are no per-cpu pages on most CPUs
> (and there can easily be hundreds of them).
> 
> This also has the side effect of shortening the average
> latency of direct reclaim by 1 or more order of magnitude
> since waiting for all the CPUs to ACK the IPI takes a
> long time.
> 
> Tested by running "hackbench 400" on a 4 CPU x86 otherwise
> idle VM and observing the difference between the number
> of direct reclaim attempts that end up in drain_all_pages()
> and those were more then 1/2 of the online CPU had any
> per-cpu page in them, using the vmstat counters introduced
> in the next patch in the series and using proc/interrupts.
> 
> In the test sceanrio, this saved around 500 global IPIs.
> After trigerring an OOM:
> 
> $ cat /proc/vmstat
> ...
> pcp_global_drain 627
> pcp_global_ipi_saved 578
> 
> I've also seen the number of drains reach 15k calls
> with the saved percentage reaching 99% when there
> are more tasks running during an OOM kill.
> 
> Signed-off-by: Gilad Ben-Yossef<gilad@...yossef.com>
> Acked-by: Christoph Lameter<cl@...ux.com>
> CC: Chris Metcalf<cmetcalf@...era.com>
> CC: Peter Zijlstra<a.p.zijlstra@...llo.nl>
> CC: Frederic Weisbecker<fweisbec@...il.com>
> CC: Russell King<linux@....linux.org.uk>
> CC: linux-mm@...ck.org
> CC: Pekka Enberg<penberg@...nel.org>
> CC: Matt Mackall<mpm@...enic.com>
> CC: Sasha Levin<levinsasha928@...il.com>
> CC: Rik van Riel<riel@...hat.com>
> CC: Andi Kleen<andi@...stfloor.org>
> CC: Mel Gorman<mel@....ul.ie>
> CC: Andrew Morton<akpm@...ux-foundation.org>
> CC: Alexander Viro<viro@...iv.linux.org.uk>
> CC: linux-fsdevel@...r.kernel.org
> CC: Avi Kivity<avi@...hat.com>
> ---
>   Christopth Ack was for a previous version that allocated
>   the cpumask in drain_all_pages().

When you changed a patch design and implementation, ACKs are
should be dropped. otherwise you miss to chance to get a good
review.



>   mm/page_alloc.c |   26 +++++++++++++++++++++++++-
>   1 files changed, 25 insertions(+), 1 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 2b8ba3a..092c331 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -67,6 +67,14 @@ DEFINE_PER_CPU(int, numa_node);
>   EXPORT_PER_CPU_SYMBOL(numa_node);
>   #endif
> 
> +/*
> + * A global cpumask of CPUs with per-cpu pages that gets
> + * recomputed on each drain. We use a global cpumask
> + * for to avoid allocation on direct reclaim code path
> + * for CONFIG_CPUMASK_OFFSTACK=y
> + */
> +static cpumask_var_t cpus_with_pcps;
> +
>   #ifdef CONFIG_HAVE_MEMORYLESS_NODES
>   /*
>    * N.B., Do NOT reference the '_numa_mem_' per cpu variable directly.
> @@ -1119,7 +1127,19 @@ void drain_local_pages(void *arg)
>    */
>   void drain_all_pages(void)
>   {
> -	on_each_cpu(drain_local_pages, NULL, 1);
> +	int cpu;
> +	struct per_cpu_pageset *pcp;
> +	struct zone *zone;
> +

get_online_cpu() ?

> +	for_each_online_cpu(cpu)
> +		for_each_populated_zone(zone) {
> +			pcp = per_cpu_ptr(zone->pageset, cpu);
> +			if (pcp->pcp.count)
> +				cpumask_set_cpu(cpu, cpus_with_pcps);
> +			else
> +				cpumask_clear_cpu(cpu, cpus_with_pcps);

cpumask* functions can't be used locklessly?

> +		}
> +	on_each_cpu_mask(cpus_with_pcps, drain_local_pages, NULL, 1);
>   }
> 
>   #ifdef CONFIG_HIBERNATION
> @@ -3623,6 +3643,10 @@ static void setup_zone_pageset(struct zone *zone)
>   void __init setup_per_cpu_pageset(void)
>   {
>   	struct zone *zone;
> +	int ret;
> +
> +	ret = zalloc_cpumask_var(&cpus_with_pcps, GFP_KERNEL);
> +	BUG_ON(!ret);
> 
>   	for_each_populated_zone(zone)
>   		setup_zone_pageset(zone);

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/