linux-kernel - Re: Do we need to call calibrate_delay() for all cores/hyperthreads on a socket?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20101203150928.GG3344@sgi.com>
Date:	Fri, 3 Dec 2010 09:09:29 -0600
From:	Robin Holt <holt@....com>
To:	Andi Kleen <andi@...stfloor.org>
Cc:	Robin Holt <holt@....com>, linux-kernel@...r.kernel.org,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Adrian Bunk <bunk@...nel.org>,
	Venkatesh Pallipadi <venkatesh.pallipadi@...el.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>, Jack Steiner <steiner@....com>
Subject: Re: Do we need to call calibrate_delay() for all
 cores/hyperthreads on a socket?

On Fri, Dec 03, 2010 at 10:36:18AM +0100, Andi Kleen wrote:
> On Fri, Dec 03, 2010 at 12:52:12AM -0600, Robin Holt wrote:
> > 
> > I do not know all the different combinations of sockets, cores, and
> > hyperthreads out there, but it seems like all cores and their hyperthreads
> > on a socket should compute the same value for their calculate_delay()
> > function.
> > 
> > When booting a 4096 cpu system without specifying lpj on the command line,
> > we spend approximately 0.1 seconds per core/hyperthread calculating the
> > lpj value for that cpu.
> > 
> > If we were to, on the other hand, only calculate the delay value for the
> > first core on a socket, we would reduce the time spent booting a 4096 cpu
> > (256 sockets, 8 cores per socket hyperthreaded) down from nearly seven
> > minutes to approx 25 seconds.  This seems like a very safe optimization,
> > but I repeat that I do not know all the different potential combinations
> > of socket, core, hyperthread out that.  Please note these are just rough
> > approximations taken from memory.  I am doing a couple of test boots
> > now without and with lpj= specified on the command line to get a more
> > accurate approximation.
> 
> Hi Robin,
> 
> At least for SMT threads that seems like a fine idea: in general I think
> we should be doing a lot more things "per core" instead of "per thread".
> 
> As for sharing calibration on whole sockets: on some systems individual 
> cores on a socket can run with different frequencies (e.g. AMD Fam10h).
> In theory the CPU should always run at P0 when coming out of boot,
> but we've had cases in the past where it was running lower, e.g.
> if the BIOS decided to cap power consumption. I guess in theory
> those states could be different per core on those systems.
> 
> So I think it would be safer to still calibrate per core at least.
> In theory one could try to check for the case described above,
> and only do it then, but that may be fragile.
> 
> But maybe it's possible to calibrate different cores in a socket in 
> parallel anyways?


Ok, more accurate timing information.

Booted without lpj=:
[   95.105108] Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 Ok.
...
[  411.231708] Booting Node 255, Processors  #4088 #4089 #4090 #4091 #4092 #4093 #4094 #4095 Ok.

Booted with lpj=:
[   95.327867] Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 Ok.
...
[  168.246617] Booting Node 255, Processors  #4088 #4089 #4090 #4091 #4092 #4093 #4094 #4095 Ok.

That is not a fair comparison due to the lpj case, we did no calibrations,
but it is illustrative.  In that time span, on the first boot we performed
4095 calibrate_delay() calls which are not performed in the second boot.
The difference is 243 seconds or 0.06 seconds per cpu.  As Jack likes
to jokingly remind me, we have once again proved that if you multiply
a small number by a large enough number, you will get a large number.

If I am calculating this correctly, just doing the calibrate_delay on
a per-socket basis should reduce boot time 228 seconds.

If, on the other hand, we do it per core, it ends up only reducing boot
time by 121 seconds.  An improvement for sure, but not good enough to
make me really happy ;).


Thanks,
Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/