linux-kernel - Re: [PATCH 0/13] Parallel struct page initialisation v4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1430410227.8193.0@cpanel21.proisp.no>
Date:	Fri, 01 May 2015 00:10:27 +0800
From:	Daniel J Blueman <daniel@...ascale.com>
To:	nzimmer <nzimmer@....com>, Mel Gorman <mgorman@...e.de>
Cc:	Pekka Enberg <penberg@...nel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Dave Hansen <dave.hansen@...el.com>,
	Waiman Long <waiman.long@...com>,
	Scott Norton <scott.norton@...com>,
	Linux-MM <linux-mm@...ck.org>,
	LKML <linux-kernel@...r.kernel.org>,
	'Steffen Persvold' <sp@...ascale.com>
Subject: Re: [PATCH 0/13] Parallel struct page initialisation v4

On Wed, Apr 29, 2015 at 2:38 AM, nzimmer <nzimmer@....com> wrote:
> On 04/28/2015 11:06 AM, Pekka Enberg wrote:
>> On Tue, Apr 28, 2015 at 5:36 PM, Mel Gorman <mgorman@...e.de> wrote:
>>> Struct page initialisation had been identified as one of the 
>>> reasons why
>>> large machines take a long time to boot. Patches were posted a long 
>>> time ago
>>> to defer initialisation until they were first used.  This was 
>>> rejected on
>>> the grounds it should not be necessary to hurt the fast paths. This 
>>> series
>>> reuses much of the work from that time but defers the 
>>> initialisation of
>>> memory to kswapd so that one thread per node initialises memory 
>>> local to
>>> that node.
>>> 
>>> After applying the series and setting the appropriate Kconfig 
>>> variable I
>>> see this in the boot log on a 64G machine
>>> 
>>> [    7.383764] kswapd 0 initialised deferred memory in 188ms
>>> [    7.404253] kswapd 1 initialised deferred memory in 208ms
>>> [    7.411044] kswapd 3 initialised deferred memory in 216ms
>>> [    7.411551] kswapd 2 initialised deferred memory in 216ms
>>> 
>>> On a 1TB machine, I see
>>> 
>>> [    8.406511] kswapd 3 initialised deferred memory in 1116ms
>>> [    8.428518] kswapd 1 initialised deferred memory in 1140ms
>>> [    8.435977] kswapd 0 initialised deferred memory in 1148ms
>>> [    8.437416] kswapd 2 initialised deferred memory in 1148ms
>>> 
>>> Once booted the machine appears to work as normal. Boot times were 
>>> measured
>>> from the time shutdown was called until ssh was available again.  
>>> In the
>>> 64G case, the boot time savings are negligible. On the 1TB machine, 
>>> the
>>> savings were 16 seconds.

> On an older 8 TB box with lots and lots of cpus the boot time, as 
> measure from grub to login prompt, the boot time improved from 1484 
> seconds to exactly 1000 seconds.
> 
> I have time on 16 TB box tonight and a 12 TB box thursday and will 
> hopefully have more numbers then.

Neat, and a roughly similar picture here.

On a 7TB, 1728-core NumaConnect system with 108 NUMA nodes, we're 
seeing stock 4.0 boot in 7136s. This drops to 2159s, or a 70% reduction 
with this patchset. Non-temporal PMD init [1] drops this to 1045s.

Nathan, what do you guys see with the non-temporal PMD patch [1]? Do 
add a sfence at the ende label if you manually patch.

Thanks!
  Daniel

[1] https://lkml.org/lkml/2015/4/23/350

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/