linux-kernel - Re: [PATCH] memory hotplug disable boot option

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20100628110658.387B.A69D9226@jp.fujitsu.com>
Date:	Mon, 28 Jun 2010 11:20:27 +0900 (JST)
From:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>
To:	Nathan Fontenot <nfont@...tin.ibm.com>
Cc:	kosaki.motohiro@...fujitsu.com, Andi Kleen <andi@...stfloor.org>,
	linux-kernel@...r.kernel.org, Greg Kroah-Hartman <gregkh@...e.de>,
	"Eric W. Biederman" <ebiederm@...ssion.com>
Subject: Re: [PATCH] memory hotplug disable boot option

> On 06/25/2010 04:19 AM, Andi Kleen wrote:
> > KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com> writes:
> > 
> >>> Proposed patch to disable memory hotplug via a boot option,
> >>> mem_hotplug=[on|off].  The patch only disables memory hotplug in that it 
> >>> prevents the creation of the memory sysfs directories for memory sections.
> >>>
> >>> This patch is meant to help alleviate very long boot times on systems with
> >>> large memory (1+ TB) and many memory sections (10's of thousands).
> >>
> >> Why making simple /sys file is so slowly? Couldn't we fix such performance
> >> problem?
> 
> The issue is the large number of sysfs memory directories that get created.
> On a system with 1 TB of memory I am seeing ~63,00 directories.  The long
> creation time is due to the string compare check in sysfs code to ensure
> we are not creating a directory with a duplicate name.

Ah, I see. probably this is sysfs issue. So Let's cc Greg and Eric.
Greg, I have dumb question. Why sysfs call strcmp() so heavily? I mean why sysfs
don't have hash based name dupliation check?

 - kosaki


> > 
> > Yes I agree this really needs to be fixed properly, not hacked around
> > with an option. Nathan can you please post some profile logs of the long
> > boot times?  
> > 
> 
> At this point the only profiling data I have is from adding a printk before
> and after the creation of the memory sysfs files in drivers/base/memory.c and
> booting with printk.time=1.
> 
> With 250 GB of memory: 10 seconds
> [    0.539562] Memory Start
> [   10.450409] Memory End
> 
> With 1 TB of memory: 9.1 minutes
> [   31.680168] Memory Start
> [  584.186500] Memory End
> 
> I am hoping to get access to a machine with 2 TB of memory sometime soon and
> can post data for boot times on an unpatched kernel.
> 
> I posted a patch earlier that updated sysfs to use reb-black trees to store
> the sysfs dirent structs (though the patch has issues with namespaces).  Using
> this patch on a 1 TB system the memory sysfs dir creation dropped to 2.2 minutes.
> 
> [    1.295874] Memory Start
> [  137.293510] Memory End
> 
> On a system with 2 TB of memory, the reb-black tree patch dropped the sysfs file
> creation time down to 33 minutes and total boot time to 1 hour 15 minutes. I
> haven't measured an unpatched kernel, but total boot of an unpatched kernel
> is just over 8 hours.
> 
> With 2 TB and patched kernel: 33 minutes
> [    3.241679] Memory Start
> [ 1986.973324] Memory End
>  
> I am open to other ideas on a solution for this.  With this many memory sections
> I feel that the sysfs memory directory really isn't human readable, not with
> 10's or 100's of thousands of entries.  Perhaps moving to a flat file representation
> and not create all the directories?
> 
> -Nathan


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/