linux-kernel - Re: [RFC PATCH v3 08/31] KVM: selftests: Require GCC to realign stacks on function entry

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <diqzlekzkazq.fsf@ackerleytng-cloudtop.c.googlers.com>
Date:   Wed, 15 Feb 2023 00:50:33 +0000
From:   Ackerley Tng <ackerleytng@...gle.com>
To:     Sean Christopherson <seanjc@...gle.com>
Cc:     erdemaktas@...gle.com, mail@...iej.szmigiero.name,
        linux-kselftest@...r.kernel.org, pbonzini@...hat.com,
        isaku.yamahata@...el.com, sagis@...gle.com, afranji@...gle.com,
        runanwang@...gle.com, shuah@...nel.org, drjones@...hat.com,
        maz@...nel.org, bgardon@...gle.com, jmattson@...gle.com,
        dmatlack@...gle.com, peterx@...hat.com, oupton@...gle.com,
        ricarkol@...gle.com, yang.zhong@...el.com, wei.w.wang@...el.com,
        xiaoyao.li@...el.com, pgonda@...gle.com, marcorr@...gle.com,
        eesposit@...hat.com, borntraeger@...ibm.com, eric.auger@...hat.com,
        wangyanan55@...wei.com, aaronlewis@...gle.com, vkuznets@...hat.com,
        pshier@...gle.com, axelrasmussen@...gle.com,
        zhenzhong.duan@...el.com, like.xu@...ux.intel.com,
        linux-kernel@...r.kernel.org, kvm@...r.kernel.org
Subject: Re: [RFC PATCH v3 08/31] KVM: selftests: Require GCC to realign
 stacks on function entry


> On Mon, Jan 23, 2023, Erdem Aktas wrote:
> > On Mon, Jan 23, 2023 at 10:53 AM Sean Christopherson  
> <seanjc@...gle.com> wrote:
> > >
> > > On Mon, Jan 23, 2023, Maciej S. Szmigiero wrote:
> > > > On 23.01.2023 19:30, Erdem Aktas wrote:
> > > > > On Fri, Jan 20, 2023 at 4:28 PM Sean Christopherson  
> <seanjc@...gle.com> wrote:
> > > > > >
> > > > > > On Sat, Jan 21, 2023, Ackerley Tng wrote:
> > > > > > > Some SSE instructions assume a 16-byte aligned stack, and GCC  
> compiles
> > > > > > > assuming the stack is aligned:
> > > > > > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838. This  
> combination
> > > > > > > results in a #GP in guests.
> > > > > > >
> > > > > > > Adding this compiler flag will generate an alternate prologue  
> and
> > > > > > > epilogue to realign the runtime stack, which makes selftest  
> code
> > > > > > > slower and bigger, but this is okay since we do not need  
> selftest code
> > > > > > > to be extremely performant.
> > > > > >
> > > > > > Huh, I had completely forgotten that this is why SSE is  
> problematic.  I ran into
> > > > > > this with the base UPM selftests and just disabled SSE.   
> /facepalm.
> > > > > >
> > > > > > We should figure out exactly what is causing a misaligned  
> stack.  As you've noted,
> > > > > > the x86-64 ABI requires a 16-byte aligned RSP.  Unless I'm  
> misreading vm_arch_vcpu_add(),
> > > > > > the starting stack should be page aligned, which means  
> something is causing the
> > > > > > stack to become unaligned at runtime.  I'd rather hunt down  
> that something than
> > > > > > paper over it by having the compiler force realignment.
> > > > >
> > > > > Is not it due to the 32bit execution part of the guest code at  
> boot
> > > > > time. Any push/pop of 32bit registers might make it a 16-byte
> > > > > unaligned stack.
> > > >
> > > > 32-bit stack needs to be 16-byte aligned, too (at function call  
> boundaries) -
> > > > see [1] chapter 2.2.2 "The Stack Frame"
> > >
> > > And this showing up in the non-TDX selftests rules that out as the  
> sole problem;
> > > the selftests stuff 64-bit mode, i.e. don't have 32-bit boot code.
> >
> > Thanks Maciej and Sean for the clarification. I was suspecting the
> > hand-coded assembly part that we have for TDX tests but  it being
> > happening in the non-TDX selftests disproves it.

> Not necessarily, it could be both.  Goofs in the handcoded assembly and  
> PEBKAC
> on my end :-)

I figured it out!

GCC assumes that the stack is 16-byte aligned **before** the call
instruction. Since call pushes rip to the stack, GCC will compile code
assuming that on entrance to the function, the stack is -8 from a
16-byte aligned address.

Since for TDs we do a ljmp to guest code, providing a function's
address, the stack was not modified by a call instruction pushing rip to
the stack, so the stack is 16-byte aligned when the guest code starts
running, instead of 16-byte aligned -8 that GCC expects.

For VMs, we set rip to a function pointer, and the VM starts running
with a 16-byte algined stack too.

To fix this, I propose that in vm_arch_vcpu_add(), we align the
allocated stack address and then subtract 8 from that:

@@ -573,10 +573,13 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm,  
uint32_t vcpu_id,
         vcpu_init_cpuid(vcpu, kvm_get_supported_cpuid());
         vcpu_setup(vm, vcpu);

+       stack_vaddr += (DEFAULT_STACK_PGS * getpagesize());
+       stack_vaddr = ALIGN_DOWN(stack_vaddr, 16) - 8;
+
         /* Setup guest general purpose registers */
         vcpu_regs_get(vcpu, &regs);
         regs.rflags = regs.rflags | 0x2;
-       regs.rsp = stack_vaddr + (DEFAULT_STACK_PGS * getpagesize());
+       regs.rsp = stack_vaddr;
         regs.rip = (unsigned long) guest_code;
         vcpu_regs_set(vcpu, &regs);