linux-kernel - Re: [PATCH v3 1/2] x86/AMD: Fix cpu_llc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20161107155626.rjapdlgiredm7uvh@pd.tnic>
Date:   Mon, 7 Nov 2016 16:56:26 +0100
From:   Borislav Petkov <bp@...e.de>
To:     Ingo Molnar <mingo@...nel.org>
Cc:     x86@...nel.org, Yazen Ghannam <Yazen.Ghannam@....com>,
        linux-kernel@...r.kernel.org,
        Peter Zijlstra <a.p.zijlstra@...llo.nl>
Subject: Re: [PATCH v3 1/2] x86/AMD: Fix cpu_llc_id for AMD Fam17h systems

On Mon, Nov 07, 2016 at 03:07:46PM +0100, Ingo Molnar wrote:
>  - cache domains might be seriously mixed up, resulting in serious drop in
>    performance.
>
>  - or domains might be partitioned 'wrong' but not catastrophically
>   wrong, resulting in a minor performance drop (if at all)

Something between the two.

Here's some debugging output from set_cpu_sibling_map():

[    0.202033] smpboot: set_cpu_sibling_map: cpu: 0, has_smt: 0, has_mp: 1
[    0.202043] smpboot: set_cpu_sibling_map: first loop, llc(this): 65528, o: 0, llc(o): 65528
[    0.202058] smpboot: set_cpu_sibling_map: first loop, link mask smt

so we link it into the SMT mask even if has_smt is off.

[    0.202067] smpboot: set_cpu_sibling_map: first loop, link mask llc
[    0.202077] smpboot: set_cpu_sibling_map: second loop, llc(this): 65528, o: 0, llc(o): 65528
[    0.202091] smpboot: set_cpu_sibling_map: second loop, link mask die

I've attached the debug diff.

And since those llc(o), i.e. the cpu_llc_id of the *other* CPU in the
loops in set_cpu_sibling_map() underflows, we're generating the funniest
thread_siblings masks and then when I run 8 threads of nbench, they get
spread around the LLC domains in a very strange pattern which doesn't
give you the normal scheduling spread one would expect for performance.

And this is just one workload - I can't imagine what else might be
influenced by this funkiness.

Oh and other things like EDAC use cpu_llc_id so they will be b0rked too.

So we absolutely need to fix that cpu_llc_id thing.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

View attachment "set_cpu_sibling_map_debug.diff" of type "text/x-diff" (1597 bytes)