linux-kernel - Re: I915 CI-run with kfence enabled, issues found

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANpmjNO+_4C0dYs6K8Ofy-xVSYxO8OtXSRbW6vCXBYdjJSjqbQ@mail.gmail.com>
Date:   Mon, 29 Mar 2021 23:54:59 +0200
From:   Marco Elver <elver@...gle.com>
To:     Andy Lutomirski <luto@...capital.net>
Cc:     Dave Hansen <dave.hansen@...el.com>,
        "Sarvela, Tomi P" <tomi.p.sarvela@...el.com>,
        kasan-dev <kasan-dev@...glegroups.com>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        Andy Lutomirski <luto@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        "the arch/x86 maintainers" <x86@...nel.org>,
        "H. Peter Anvin" <hpa@...or.com>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: I915 CI-run with kfence enabled, issues found

On Mon, 29 Mar 2021 at 23:47, Andy Lutomirski <luto@...capital.net> wrote:
>
>
> > On Mar 29, 2021, at 2:34 PM, Marco Elver <elver@...gle.com> wrote:
> >
> > On Mon, 29 Mar 2021 at 23:03, Dave Hansen <dave.hansen@...el.com> wrote:
> >>> On 3/29/21 10:45 AM, Marco Elver wrote:
> >>>> On Mon, 29 Mar 2021 at 19:32, Dave Hansen <dave.hansen@...el.com> wrote:
> >>> Doing it to all CPUs is too expensive, and we can tolerate this being
> >>> approximate (nothing bad will happen, KFENCE might just miss a bug and
> >>> that's ok).
> >> ...
> >>>> BTW, the preempt checks in flush_tlb_one_kernel() are dependent on KPTI
> >>>> being enabled.  That's probably why you don't see this everywhere.  We
> >>>> should probably have unconditional preempt checks in there.
> >>>
> >>> In which case I'll add a preempt_disable/enable() pair to
> >>> kfence_protect_page() in arch/x86/include/asm/kfence.h.
> >>
> >> That sounds sane to me.  I'd just plead that the special situation (not
> >> needing deterministic TLB flushes) is obvious.  We don't want any folks
> >> copying this code.
> >>
> >> BTW, I know you want to avoid the cost of IPIs, but have you considered
> >> any other low-cost ways to get quicker TLB flushes?  For instance, you
> >> could loop over all CPUs and set cpu_tlbstate.invalidate_other=1.  That
> >> would induce a context switch at the next context switch without needing
> >> an IPI.
> >
> > This is interesting. And it seems like it would work well for our
> > usecase. Ideally we should only flush entries related to the page we
> > changed. But it seems invalidate_other would flush the entire TLB.
> >
> > With PTI, flush_tlb_one_kernel() already does that for the current
> > CPU, but now we'd flush entire TLBs for all CPUs and even if PTI is
> > off.
> >
> > Do you have an intuition for how much this would affect large
> > multi-socket systems? I currently can't quite say, and would err on
> > the side of caution.
>
> Flushing the kernel TLB for all addresses
> Is rather pricy. ISTR 600 cycles on Skylake, not to mention the cost of losing the TLB.  How common is this?

AFAIK, invalidate_other resets the asid, so it's not explicit and
perhaps cheaper?

In any case, if we were to do this, it'd be based on the sample
interval of KFENCE, which can be as low as 1ms. But this is a
production debugging feature, so the target machines are not test
machines. For those production deployments we'd be looking at every
~500ms. But I know of other deployments that use <100ms.

Doesn't sound like much, but as you say, I also worry a bit about
losing the TLB across >100 CPUs even if it's every 500ms.

Thanks,
-- Marco