linux-kernel - Re: gcc-10: kernel stack is corrupted and fails to boot

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK8P3a1dxJAHCZ19=sPUkDi5wLWeJ6KKtD09Wmjqkz27TQN6Xw@mail.gmail.com>
Date:   Wed, 13 May 2020 10:49:31 +0200
From:   Arnd Bergmann <arnd@...db.de>
To:     Kalle Valo <kvalo@...eaurora.org>
Cc:     linux-wireless <linux-wireless@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: gcc-10: kernel stack is corrupted and fails to boot

On Wed, May 13, 2020 at 8:50 AM Kalle Valo <kvalo@...eaurora.org> wrote:
>
> Kalle Valo <kvalo@...eaurora.org> writes:
>
> > This motivated me to switch to using GCC 10.x and I noticed that you had
> > already upgraded crosstool so it was a trivial thing to do, awesome :)
> >
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/
>
> And now I have a problem :) I first noticed that my x86 testbox is not
> booting when I compile the kernel with GCC 10.1.0 from crosstool. I
> didn't get any error messages so I just downgraded the compiler and the
> kernel was booting fine again. Next I decided to try GCC 10.1 with my
> x86 laptop and it also failed to boot, but this time I got kernel logs
> and saw this:
>
> Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: start_secodary+0x178/0x180
>
> Call Trace:
> dump_stack
> panic
> ? _raw_spin_unlock_irqrestore
> ? start_secondary
> __stack_chk_fail
> start_secondary
> secondary_startup
>
> (I wrote the above messages manually from a picture so expect typos)
>
> Then also on my x86 laptop I downgraded the compiler to GCC 8.1.0 (from
> crosstool), rebuilt the exactly same kernel version and the kernel
> booted without issues.
>
> I'm using 5.7.0-rc4-wt-ath+ which is basically v5.7-rc4 plus latest
> wireless patches, and I doubt the wireless patches are making any
> difference this early in the boot. All compilers I use are prebuilt
> binaries from kernel.org crosstool repo[1] with addition of ccache
> v3.4.1 to speed up my builds.
>
> Any ideas? How should I debug this further?

At least if it fails reproducibly, it's probably not too hard to drill
down further. Some ideas:

* I'd first try to reproduce it in qemu. Since you don't even need
  any user space or modules, I would simply try
  $ qemu-system-x86_64 -nographic -monitor none -append
"console=ttyS0" -serial stdio -smp 4 -kernel arch/x86/boot/bzImage
  I tried it here with an x86 defconfig linux-next kernel but did not
  run into the problem you described. If you share your .config,
  I can try reproducing with that as well. Once there is a reproducer
  in qemu, it should be trivial to step through it using gdb.

* There are still two prerelease compiler versions on kernel.org,
  from February and from April. You can try each one to see
  if this was a recent regression. It's also possible that there is
  a problem with my specific builds of gcc-10.1, and that the
  compiler is actually fine for others.The gcc-10 packages in
  Fedora/Debian/Ubuntu are probably better tested.

       Arnd