Date:   Fri, 16 Jun 2017 11:03:59 -0700
From:   "H.J. Lu" <hjl.tools@...il.com>
To:     Andy Lutomirski <luto@...nel.org>
Cc:     Dave Hansen <dave.hansen@...el.com>,
        "Robert O'Callahan" <robert@...llahan.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        X86 ML <x86@...nel.org>
Subject: Re: xgetbv nondeterminism

On Fri, Jun 16, 2017 at 10:56 AM, Andy Lutomirski <luto@...nel.org> wrote:
> On Fri, Jun 16, 2017 at 10:44 AM, H.J. Lu <hjl.tools@...il.com> wrote:
>> On Fri, Jun 16, 2017 at 9:38 AM, Andy Lutomirski <luto@...nel.org> wrote:
>>> On Fri, Jun 16, 2017 at 9:17 AM, H.J. Lu <hjl.tools@...il.com> wrote:
>>>> On Fri, Jun 16, 2017 at 9:01 AM, Andy Lutomirski <luto@...nel.org> wrote:
>>>>> On Thu, Jun 15, 2017 at 9:34 PM, H.J. Lu <hjl.tools@...il.com> wrote:
>>>>>> On Thu, Jun 15, 2017 at 8:05 PM, Andy Lutomirski <luto@...nel.org> wrote:
>>>>>>> On Thu, Jun 15, 2017 at 7:17 PM, H.J. Lu <hjl.tools@...il.com> wrote:
>>>>>>>> On Thu, Jun 15, 2017 at 4:28 PM, Andy Lutomirski <luto@...nel.org> wrote:
>>>>>>>>> On Thu, Jun 15, 2017 at 4:11 PM, H.J. Lu <hjl.tools@...il.com> wrote:
>>>>>>>>>> It is used for lazy binding the first time when an external function is called.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Maybe I'm just being dense, but why?  What does ld.so need to do to
>>>>>>>>> resolve a symbol and update the GOT that requires using extended
>>>>>>>>> state?
>>>>>>>>
>>>>>>>> Since the first 8 vector registers are used to pass function parameters
>>>>>>>> and ld.so uses vector registers, _dl_runtime_resolve needs to preserve
>>>>>>>> the first 8 vector registers when transferring control to ld.so.
>>>>>>>>
>>>>>>>
>>>>>>> Wouldn't it be faster and more future-proof to recompile the relevant
>>>>>>> parts of ld.so to avoid using extended state?
>>>>>>>
>>>>>>
>>>>>> Are you suggesting not to use vector in ld.so?
>>>>>
>>>>> Yes, exactly.
>>>>>
>>>>>>  We used to do that
>>>>>> several years ago, which leads to some subtle bugs, like
>>>>>>
>>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=15128
>>>>>
>>>>> I don't think x86_64 has the issue that ARM has there.  The Linux
>>>>> kernel, for example, has always been compiled to not use vector or
>>>>> floating point registers on x86 (32 and 64), and it works fine.  Linux
>>>>> doesn't save extended regs on kernel entry and it doesn't restore them
>>>>> on exit.
>>>>>
>>>>> I would suggest that ld.so be compiled without use of vector
>>>>> registers, that the normal lazy binding path not try to save any extra
>>>>> regs, and that ifuncs be called through a thunk that saves whatever
>>>>> registers need saving, possibly just using XSAVEOPT.  After all, ifunc
>>>>> is used for only a tiny fraction of symbols.
>>>>
>>>> x86-64 was the only target which used FOREIGN_CALL macros
>>>> in ld.so,  FOREIGN_CALL macros were the cause of race condition
>>>> in ld.so:
>>>>
>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=11214
>>>>
>>>> Not to save and restore the first 8 vector registers means that
>>>> FOREIGN_CALL macros have to be used.  We don't want to
>>>> do that on x86-64.
>>>>
>>>>
>>>
>>> You're talking about this, right:
>>>
>>> commit f3dcae82d54e5097e18e1d6ef4ff55c2ea4e621e
>>> Author: H.J. Lu <hjl.tools@...il.com>
>>> Date:   Tue Aug 25 04:33:54 2015 -0700
>>>
>>>     Save and restore vector registers in x86-64 ld.so
>>>
>>> It seems to me that the problem wasn't that the save/restore happened
>>> on some of the time -- it was that the save and restore code used a
>>> TLS variable to track its own state.  Shouldn't it have been a stack
>>> variable or even just implicit in the control flow?
>>
>> No, it can't use stack variable since _dl_runtime_resolve never
>> returns.
>
> I haven't dug all the way through the source, but surely ifuncs are
> CALLed, not JMPed to.  That means you have a stack somewhere.  This
> stuff is mostly written in C, and local variables should work just
> fine.
>
>>
>>> In any case, glibc is effectively doing a foreign call anyway, right?
>>
>> No.
>>
>>> It's doing the foreign call to itself on every lazy binding
>>> resolution, though, which seems quite expensive.  I'm saying that it
>>> seems like it would be more sensible to do the complicated foreign
>>> call logic only when doing the dangerous case, which is when lazy
>>> binding calls an ifunc.
>>>
>>> If I were to rewrite this, I would do it like this:
>>>
>>> void *call_runtime_ifunc(void (*ifunc)());  // or whatever the
>>> signature needs to be
>>
>> It is unrelated to IFUNC.  This is how external function call works.
>
> External function call to what external function?  Are there any calls
> to any non-IFUNC external functions that are triggered by runtime
> resolution?
>
> In any event, I still don't understand the issue.  The code does this,
> effectively:
>
> PLT -> GOT
> GOT points to a stub that transfers control to ld.so
> ld.so resolves the symbol (_dl_fixup, I think)
> ld.so patches the GOT
> ld.so jumps to the resolved function
>
> As far as I can tell, the only part of the whole process that might
> touch vector registers at all is elf_ifunc_invoke().  Couldn't all the
> register saving and restoring be moved to elf_ifunc_invoke()?

Please grep for  FOREIGN_CALL the elf directory.

-- 
H.J.

Powered by blists - more mailing lists