linux-kernel - Re: [PATCH v4 0/9] Parallel CPU bringup for x86

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ff876008-b642-4dbc-aa41-1639905e08b6@bytedance.com>
Date:   Wed, 1 Feb 2023 14:40:09 +0000
From:   Usama Arif <usama.arif@...edance.com>
To:     David Woodhouse <dwmw2@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>
Cc:     Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        Dave Hansen <dave.hansen@...ux.intel.com>, x86@...nel.org,
        "H . Peter Anvin" <hpa@...or.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        "Paul E . McKenney" <paulmck@...nel.org>,
        linux-kernel@...r.kernel.org, kvm@...r.kernel.org,
        rcu@...r.kernel.org, mimoja@...oja.de, hewenliang4@...wei.com,
        hushiyuan@...wei.com, luolongjun@...wei.com, hejingxian@...wei.com,
        Tom Lendacky <thomas.lendacky@....com>,
        Sean Christopherson <seanjc@...gle.com>,
        Paul Menzel <pmenzel@...gen.mpg.de>,
        Fam Zheng <fam.zheng@...edance.com>,
        Punit Agrawal <punit.agrawal@...edance.com>,
        simon.evans@...edance.com, liangma@...ngbit.com
Subject: Re: [PATCH v4 0/9] Parallel CPU bringup for x86_64



On 01/02/2022 20:53, David Woodhouse wrote:
> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for
> them shaves about 80% off the AP bringup time on a 96-thread 2-socket
> Skylake box (EC2 c5.metal) — from about 500ms to 100ms.
> 
> There are more wins to be had with further parallelisation, but this is
> the simple part.
> 

Hi,

We are interested in reducing the boot time of servers (with kexec), and 
smpboot takes up a significant amount of time while booting. When 
testing the patch series (rebased to v6.1) on a server with 128 CPUs 
split across 2 NUMA nodes, it brought down the smpboot time from ~700ms 
to 100ms. Adding another cpuhp state for do_wait_cpu_initialized to make 
sure cpu_init is reached (as done in v1 of the series + using the 
cpu_finishup_mask) brought it down further to ~30ms.

I just wanted to check what was needed to progress the patch series 
further for review? There weren't any comments on v4 of the patch so I 
couldn't figure out what more is needed. I think its quite useful to 
have this working so would be really glad help in anything needed to 
restart the review.

Thanks!
Usama



> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
>      in preparation for more parallelisation.
> v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
>      avoid scribbling on initial_gs in common_cpu_up(), and to allow all
>      24 bits of the physical X2APIC ID to be used. That patch still needs
>      a Signed-off-by from its original author, who once claimed not to
>      remember writing it at all. But now we've fixed it, hopefully he'll
>      admit it now :)
> 
> David Woodhouse (8):
>        x86/apic/x2apic: Fix parallel handling of cluster_mask
>        cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
>        cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU
>        x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
>        x86/smpboot: Split up native_cpu_up into separate phases and document them
>        x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
>        x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup
>        x86/smpboot: Serialize topology updates for secondary bringup
> 
> Thomas Gleixner (1):
>        x86/smpboot: Support parallel startup of secondary CPUs
> 
> [dwoodhou@i7 linux-2.6]$ git diff --stat  v5.17-rc2..share/parallel-5.17-part1
>   arch/x86/include/asm/realmode.h       |   3 +
>   arch/x86/include/asm/smp.h            |  13 +-
>   arch/x86/include/asm/topology.h       |   2 -
>   arch/x86/kernel/acpi/sleep.c          |   1 +
>   arch/x86/kernel/apic/apic.c           |   2 +-
>   arch/x86/kernel/apic/x2apic_cluster.c | 108 ++++++-----
>   arch/x86/kernel/cpu/common.c          |   6 +-
>   arch/x86/kernel/cpu/mtrr/mtrr.c       |   9 +
>   arch/x86/kernel/head_64.S             |  73 ++++++++
>   arch/x86/kernel/smpboot.c             | 325 ++++++++++++++++++++++++----------
>   arch/x86/realmode/init.c              |   3 +
>   arch/x86/realmode/rm/trampoline_64.S  |  14 ++
>   arch/x86/xen/smp_pv.c                 |   4 +-
>   include/linux/cpuhotplug.h            |   2 +
>   include/linux/smpboot.h               |   7 +
>   kernel/cpu.c                          |  27 ++-
>   kernel/smpboot.c                      |   2 +-
>   kernel/smpboot.h                      |   2 -
>   18 files changed, 442 insertions(+), 161 deletions(-)
> 
> 
> 
>