linux-kernel - Re: [RFC 0/2] Delay initializing of large sections of memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 21 Jun 2013 14:19:18 -0700
From:	Mike Travis <travis@....com>
To:	Yinghai Lu <yinghai@...nel.org>
CC:	Nathan Zimmer <nzimmer@....com>, Robin Holt <holt@....com>,
	Rob Landley <rob@...dley.net>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
	the arch/x86 maintainers <x86@...nel.org>,
	linux-doc@...r.kernel.org,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [RFC 0/2] Delay initializing of large sections of memory



On 6/21/2013 11:36 AM, Yinghai Lu wrote:
> On Fri, Jun 21, 2013 at 9:25 AM, Nathan Zimmer <nzimmer@....com> wrote:
>> This rfc patch set delays initializing large sections of memory until we have
>> started cpus.  This has the effect of reducing startup times on large memory
>> systems.  On 16TB it can take over an hour to boot and most of that time
>> is spent initializing memory.
> 
> One hour on system with 16T ram? BIOS or OS?

The BIOS is about 20 minutes *before* the 1+ hour.  (When we started this
UV project way back when, 8TB took over 4 hours before we threw in the towel.)
> 
> I use wall clock to check bootime on one system with 3T and 16 pcie cards,
> Linus only takes about 3m and 30 seconds from bootloader.

I can send some stats on where various delays are but most of it was in
memory initialization.  On average UV nodes carry 128 or 256G per node,
so 12 nodes would take about 3 or 4 minutes, perhaps more.
> 
> wonder if you boot delay is with so many cpu get onlined in serialized mode.

Nope.  If that was the case, delaying memory but initializing all the
cpus would not affect the time.
> 
> so can you try boot your system with "maxcpus=128" to get the boot time with
> wall clock ?

We could try if you really think it will provide any useful info.  Not sure
exactly how having memory on nodes with no active cpus will react.
> 
> Thanks
> 
> Yinghai
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/