lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 21 Aug 2020 19:53:01 -0700
From:   Andy Lutomirski <>
To:     Kyle Huey <>
Cc:     "Bae, Chang Seok" <>,
        Robert O'Callahan <>,
        Andy Lutomirski <>,
        Thomas Gleixner <>,
        Ingo Molnar <>,
        "H . Peter Anvin" <>, Andi Kleen <>,
        "Shankar, Ravi V" <>,
        LKML <>,
        "Hansen, Dave" <>
Subject: Re: [REGRESSION] x86/cpu fsgsbase breaks TLS in 32 bit rr tracees on a 64 bit system

> On Aug 21, 2020, at 2:33 PM, Kyle Huey <> wrote:
> On Fri, Aug 21, 2020 at 1:08 PM Bae, Chang Seok
> <> wrote:
>>>> On Aug 20, 2020, at 21:41, Kyle Huey <> wrote:
>>> On the x86-64 5.9-rc1 TLS is completely broken in 32 bit tracees when
>>> running under rr[0]. Booting the kernel with `nofsgsbase` fixes it and
>>> I bisected to
>>> STR:
>>> 1. Build rr from source by
>>> a. git clone
>>> b. mkdir rr/obj
>>> c. cd rr/obj
>>> d. cmake ..
>>> e. make -j16
>>> 2. Run the simple 32 bit tracee outside of rr with `./bin/simple_32`.
>>> It should print a message and exit cleanly.
>>> 3. Run it under rr with `./bin/rr ./bin/simple_32`.
>>> It should behave the same way, but with fsgsbase enabled it will
>>> segfault. The `simple_32` binary is a simple "hello world" type
>>> program but it does link to pthreads, so pre-main code attempts to
>>> access TLS variables.
>>> The interplay between 32 bit and 64 bit TLS is dark magic to me
>>> unfortunately so this is all the useful information I have.
>> As I run it and collect the ptrace logs, it starts to set FSBASE with
>> some numbers, e.g. 140632147826496, and then later attempts to set GS
>> with 99 through putreg(), not putreg32().
>> With FSGSBASE, the FS/GS base is decoupled from the selector. Andy
>> made putreg32() to retain the old behavior, fetching FS/GS base
>> according to the index, but the putreg() does not do. So, rr probably
>> relies on the old behavior as observed to setting the GS index only.
>> But it was previously considered to be okay with this comment [1]:
>>   "Our modifications to fs/gs/fs_base/gs_base are always either a)
>>    setting values that the kernel set during recording to make them
>>    happen during replay or b) emulating PTRACE_SET_REGS that a tracee
>>    ptracer tried to set on another tracee. Either way I think the
>>    effects are going to be the same as what would happen if the
>>    program were run without rr."
>> It is not straightforward to go into the rr internal to me. Robert,
>> any thought?
> Hmm. When we are running a 32 bit tracee in a 64 bit build of rr we
> internally convert between the 32 bit and 64 bit user_regs_structs[0]
> at the ptrace boundary. This does not preserve the fs/gsbase (because
> there is no fs/gsbase in the 32 bit user_regs_struct, of course).
> 40c45904f818c1f6555294ca27afc5fda4f09e68 added magic for a 32 bit
> tracer tracing a 32 bit tracee on a 64 bit kernel, but it looks like a
> 64 bit tracer tracing a 32 bit tracee on a 64 bit kernel *is* now
> expected to preserve the fs/gsbase values (or die, in our case).
> Is that correct?

I was certainly not expecting rr to do this, and I thought I had asked in advance.  What exact ptrace() calls are you doing here?  Is this POKEUSER or something else?  Breaking rr is at least impolite, and I’d like to fix this.

> - Kyle
> [0]

Powered by blists - more mailing lists