linux-kernel - Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on a 64 bit system

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <29FD0DDA-3093-46A3-BCF4-85DEC229E30D@intel.com>
Date:   Fri, 21 Aug 2020 20:08:54 +0000
From:   "Bae, Chang Seok" <chang.seok.bae@...el.com>
To:     Kyle Huey <me@...ehuey.com>,
        Robert O'Callahan <rocallahan@...il.com>
CC:     Andy Lutomirski <luto@...nel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>,
        "H . Peter Anvin" <hpa@...or.com>, Andi Kleen <ak@...ux.intel.com>,
        "Shankar, Ravi V" <ravi.v.shankar@...el.com>,
        LKML <linux-kernel@...r.kernel.org>,
        "Hansen, Dave" <dave.hansen@...el.com>
Subject: Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on
 a 64 bit system


> On Aug 20, 2020, at 21:41, Kyle Huey <me@...ehuey.com> wrote:
> 
> On the x86-64 5.9-rc1 TLS is completely broken in 32 bit tracees when
> running under rr[0]. Booting the kernel with `nofsgsbase` fixes it and
> I bisected to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=673903495c85137791d5820d690229efe09c8f7b.
> 
> STR:
> 1. Build rr from source by
>  a. git clone https://github.com/mozilla/rr
>  b. mkdir rr/obj
>  c. cd rr/obj
>  d. cmake ..
>  e. make -j16
> 2. Run the simple 32 bit tracee outside of rr with `./bin/simple_32`.
> It should print a message and exit cleanly.
> 3. Run it under rr with `./bin/rr ./bin/simple_32`.
> 
> It should behave the same way, but with fsgsbase enabled it will
> segfault. The `simple_32` binary is a simple "hello world" type
> program but it does link to pthreads, so pre-main code attempts to
> access TLS variables.
> 
> The interplay between 32 bit and 64 bit TLS is dark magic to me
> unfortunately so this is all the useful information I have.

As I run it and collect the ptrace logs, it starts to set FSBASE with
some numbers, e.g. 140632147826496, and then later attempts to set GS
with 99 through putreg(), not putreg32().

With FSGSBASE, the FS/GS base is decoupled from the selector. Andy
made putreg32() to retain the old behavior, fetching FS/GS base
according to the index, but the putreg() does not do. So, rr probably
relies on the old behavior as observed to setting the GS index only.
But it was previously considered to be okay with this comment [1]:

   "Our modifications to fs/gs/fs_base/gs_base are always either a)
    setting values that the kernel set during recording to make them
    happen during replay or b) emulating PTRACE_SET_REGS that a tracee
    ptracer tried to set on another tracee. Either way I think the
    effects are going to be the same as what would happen if the
    program were run without rr."

It is not straightforward to go into the rr internal to me. Robert, 
any thought?

[1] https://mail.mozilla.org/pipermail/rr-dev/2018-March/000615.html

> - Kyle
> 
> [0] https://rr-project.org/