linux-kernel - Re: [PATCH RFC 00/10] KFENCE: A low-overhead sampling-based memory safety error detector

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANpmjNNWOVi317bF_E=QdcSSd5x6Dfk=+nECA9VnZSLGMKigYQ@mail.gmail.com>
Date:   Fri, 11 Sep 2020 15:33:44 +0200
From:   Marco Elver <elver@...gle.com>
To:     Dmitry Vyukov <dvyukov@...gle.com>
Cc:     Vlastimil Babka <vbabka@...e.cz>,
        Dave Hansen <dave.hansen@...el.com>,
        Alexander Potapenko <glider@...gle.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Catalin Marinas <catalin.marinas@....com>,
        Christoph Lameter <cl@...ux.com>,
        David Rientjes <rientjes@...gle.com>,
        Joonsoo Kim <iamjoonsoo.kim@....com>,
        Mark Rutland <mark.rutland@....com>,
        Pekka Enberg <penberg@...nel.org>,
        "H. Peter Anvin" <hpa@...or.com>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        Andrey Konovalov <andreyknvl@...gle.com>,
        Andrey Ryabinin <aryabinin@...tuozzo.com>,
        Andy Lutomirski <luto@...nel.org>,
        Borislav Petkov <bp@...en8.de>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        Eric Dumazet <edumazet@...gle.com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Ingo Molnar <mingo@...hat.com>, Jann Horn <jannh@...gle.com>,
        Jonathan Corbet <corbet@....net>,
        Kees Cook <keescook@...omium.org>,
        Peter Zijlstra <peterz@...radead.org>, Qian Cai <cai@....pw>,
        Thomas Gleixner <tglx@...utronix.de>,
        Will Deacon <will@...nel.org>,
        "the arch/x86 maintainers" <x86@...nel.org>,
        "open list:DOCUMENTATION" <linux-doc@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        kasan-dev <kasan-dev@...glegroups.com>,
        Linux ARM <linux-arm-kernel@...ts.infradead.org>,
        Linux-MM <linux-mm@...ck.org>
Subject: Re: [PATCH RFC 00/10] KFENCE: A low-overhead sampling-based memory
 safety error detector

On Fri, 11 Sep 2020 at 15:10, Dmitry Vyukov <dvyukov@...gle.com> wrote:
> On Fri, Sep 11, 2020 at 2:03 PM Marco Elver <elver@...gle.com> wrote:
> > On Fri, 11 Sep 2020 at 09:36, Dmitry Vyukov <dvyukov@...gle.com> wrote:
[...]
> > > By "reasonable" I mean if the pool will last long enough to still
> > > sample something after hours/days? Have you tried any experiments with
> > > some workload (both short-lived processes and long-lived
> > > processes/namespaces) capturing state of the pool? It can make sense
> > > to do to better understand dynamics. I suspect that the rate may need
> > > to be orders of magnitude lower.
> >
> > Yes, the current default sample interval is a lower bound, and is also
> > a reasonable default for testing. I expect real deployments to use
> > much higher sample intervals (lower rate).
> >
> > So here's some data (with CONFIG_KFENCE_NUM_OBJECTS=1000, so that
> > allocated KFENCE objects isn't artificially capped):
> >
> > -- With a mostly vanilla config + KFENCE (sample interval 100 ms),
> > after ~40 min uptime (only boot, then idle) I see ~60 KFENCE objects
> > (total allocations >600). Those aren't always the same objects, with
> > roughly ~2 allocations/frees per second.
> >
> > -- Then running sysbench I/O benchmark, KFENCE objects allocated peak
> > at 82. During the benchmark, allocations/frees per second are closer
> > to 10-15. After the benchmark, the KFENCE objects allocated remain at
> > 82, and allocations/frees per second fall back to ~2.
> >
> > -- For the same system, changing the sample interval to 1 ms (echo 1 >
> > /sys/module/kfence/parameters/sample_interval), and re-running the
> > benchmark gives me: KFENCE objects allocated peak at exactly 500, with
> > ~500 allocations/frees per second. After that, allocated KFENCE
> > objects dropped a little to 496, and allocations/frees per second fell
> > back to ~2.
> >
> > -- The long-lived objects are due to caches, and just running 'echo 1
> > > /proc/sys/vm/drop_caches' reduced allocated KFENCE objects back to
> > 45.
>
> Interesting. What type of caches is this? If there is some type of
> cache that caches particularly lots of sampled objects, we could
> potentially change the cache to release sampled objects eagerly.

The 2 major users of KFENCE objects for that workload are
'buffer_head' and 'bio-0'.

If we want to deal with those, I guess there are 2 options:

1. More complex, but more precise: make the users of them check
is_kfence_address() and release their buffers earlier.

2. Simpler, generic solution: make KFENCE stop return allocations for
non-kmalloc_caches memcaches after more than ~90% of the pool is
exhausted. This assumes that creators of long-lived objects usually
set up their own memcaches.

I'm currently inclined to go for (2).

Thanks,
-- Marco