linux-kernel - Re: [PATCH v3] x86/asm: Force native_apic_mem

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMBK9=bTMJircCK9SYHsZcKvbD0oZUByQPXcYciD6mcVqgg6=g@mail.gmail.com>
Date: Fri, 9 Feb 2024 10:20:22 -0800
From: Adam Dunlap <acdunlap@...gle.com>
To: Ard Biesheuvel <ardb@...nel.org>
Cc: Dave Hansen <dave.hansen@...el.com>, Thomas Gleixner <tglx@...utronix.de>, 
	Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>, 
	Dave Hansen <dave.hansen@...ux.intel.com>, x86@...nel.org, 
	"H. Peter Anvin" <hpa@...or.com>, Nathan Chancellor <nathan@...nel.org>, 
	Nick Desaulniers <ndesaulniers@...gle.com>, Bill Wendling <morbo@...gle.com>, 
	Justin Stitt <justinstitt@...gle.com>, "Peter Zijlstra (Intel)" <peterz@...radead.org>, 
	Arjan van de Ven <arjan@...ux.intel.com>, Wei Liu <wei.liu@...nel.org>, linux-kernel@...r.kernel.org, 
	llvm@...ts.linux.dev, Jacob Xu <jacobhxu@...gle.com>, Alper Gun <alpergun@...gle.com>, 
	Kevin Loughlin <kevinloughlin@...gle.com>, Peter Gonda <pgonda@...gle.com>
Subject: Re: [PATCH v3] x86/asm: Force native_apic_mem_read to use mov

On Fri, Feb 9, 2024 at 7:22 AM Ard Biesheuvel <ardb@...nel.org> wrote:
>
> On Thu, 8 Feb 2024 at 16:48, Dave Hansen <dave.hansen@...el.com> wrote:
> >
> > On 2/6/24 14:36, Adam Dunlap wrote:
> > ...
> > > In particular, when compiled with clang and run as a SEV-ES or
> > > SEV-SNP guest, the compiler would emit a testl instruction which is
> > > not supported by the SEV-ES emulator
> >
> > What changed?  Why is this a bug that we're only noticing now?  The line
> > of code that's modified here is from 2008.
> >
> > I assume that it's something new in clang, but it'd be great to know
> > that for sure.
> >
>
> Might be the use of LTO in the Google prod[uction]kernel. Adam, can you confirm?

It doesn't look like it's LTO. I disabled the LTO options in .config
and you can see the
problem just in a single object file:

With gcc:

% gdb -batch -ex 'file arch/x86/kernel/apic/ipi.o' -ex 'disassemble
apic_mem_wait_icr_idle'
Dump of assembler code for function apic_mem_wait_icr_idle:
   0x0000000000000260 <+0>:     endbr64
   0x0000000000000264 <+4>:     jmp    0x268 <apic_mem_wait_icr_idle+8>
   0x0000000000000266 <+6>:     pause
   0x0000000000000268 <+8>:     mov    0xffffffffff5fc300,%eax
   0x000000000000026f <+15>:    test   $0x10,%ah
   0x0000000000000272 <+18>:    jne    0x266 <apic_mem_wait_icr_idle+6>
   0x0000000000000274 <+20>:    jmpq   0x279

With clang:

% gdb -batch -ex 'file arch/x86/kernel/apic/ipi.o' -ex 'disassemble
apic_mem_wait_icr_idle'
Dump of assembler code for function apic_mem_wait_icr_idle:
   0x0000000000000250 <+0>:     endbr64
   0x0000000000000254 <+4>:     testl  $0x1000,0xffffffffff5fc300
   0x000000000000025f <+15>:    je     0x270 <apic_mem_wait_icr_idle+32>
   0x0000000000000261 <+17>:    pause
   0x0000000000000263 <+19>:    testl  $0x1000,0xffffffffff5fc300
   0x000000000000026e <+30>:    jne    0x261 <apic_mem_wait_icr_idle+17>
   0x0000000000000270 <+32>:    cs jmpq 0x276

This shows how gcc uses mov to load the apic memory and then testl to
test it, while clang
combines those instructions.

I plugged in the relevant subsection into godbolt [0] and it appears
the assembly changed in
clang 8 (released 2019). I'm not set up to do full compilations with
old clang versions, but
this is the most likely change point.

> this seems like the kind of thing we'll want in -stable in case folks
> are compiling stable kernels with new clangs.

That makes sense. Note that there was another patch accepted recently
that fixed another
clang-with-SEV problem [1], so they should probably be backported to
the same stable
branches since neither is that useful without the other.

[0] https://godbolt.org/z/nq9M9e8ex
[1] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=1c811d403afd73f04bde82b83b24c754011bd0e8