linux-kernel - Re: [PATCH v3 2/6] exit: Put an upper limit on how often we can oops

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20230119201023.4003-1-sj@kernel.org>
Date:   Thu, 19 Jan 2023 20:10:23 +0000
From:   SeongJae Park <sj@...nel.org>
To:     Kees Cook <keescook@...omium.org>
Cc:     Jann Horn <jannh@...gle.com>, Luis Chamberlain <mcgrof@...nel.org>,
        Seth Jenkins <sethjenkins@...gle.com>,
        Greg KH <gregkh@...uxfoundation.org>,
        Linus Torvalds <torvalds@...uxfoundation.org>,
        Andy Lutomirski <luto@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        tangmeng <tangmeng@...ontech.com>,
        "Guilherme G. Piccoli" <gpiccoli@...lia.com>,
        Tiezhu Yang <yangtiezhu@...ngson.cn>,
        Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        "Eric W. Biederman" <ebiederm@...ssion.com>,
        Arnd Bergmann <arnd@...db.de>,
        Dmitry Vyukov <dvyukov@...gle.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        Valentin Schneider <vschneid@...hat.com>,
        Andrey Ryabinin <ryabinin.a.a@...il.com>,
        Alexander Potapenko <glider@...gle.com>,
        Andrey Konovalov <andreyknvl@...il.com>,
        Vincenzo Frascino <vincenzo.frascino@....com>,
        David Gow <davidgow@...gle.com>,
        "Paul E. McKenney" <paulmck@...nel.org>,
        Jonathan Corbet <corbet@....net>,
        Baolin Wang <baolin.wang@...ux.alibaba.com>,
        "Jason A. Donenfeld" <Jason@...c4.com>,
        Eric Biggers <ebiggers@...gle.com>,
        Huang Ying <ying.huang@...el.com>,
        Anton Vorontsov <anton@...msg.org>,
        Mauro Carvalho Chehab <mchehab+huawei@...nel.org>,
        Laurent Dufour <ldufour@...ux.ibm.com>,
        Rob Herring <robh@...nel.org>, linux-kernel@...r.kernel.org,
        kasan-dev@...glegroups.com, linux-mm@...ck.org,
        linux-doc@...r.kernel.org, linux-hardening@...r.kernel.org
Subject: Re: [PATCH v3 2/6] exit: Put an upper limit on how often we can oops

Hello,

On Thu, 17 Nov 2022 15:43:22 -0800 Kees Cook <keescook@...omium.org> wrote:

> From: Jann Horn <jannh@...gle.com>
> 
> Many Linux systems are configured to not panic on oops; but allowing an
> attacker to oops the system **really** often can make even bugs that look
> completely unexploitable exploitable (like NULL dereferences and such) if
> each crash elevates a refcount by one or a lock is taken in read mode, and
> this causes a counter to eventually overflow.
> 
> The most interesting counters for this are 32 bits wide (like open-coded
> refcounts that don't use refcount_t). (The ldsem reader count on 32-bit
> platforms is just 16 bits, but probably nobody cares about 32-bit platforms
> that much nowadays.)
> 
> So let's panic the system if the kernel is constantly oopsing.
> 
> The speed of oopsing 2^32 times probably depends on several factors, like
> how long the stack trace is and which unwinder you're using; an empirically
> important one is whether your console is showing a graphical environment or
> a text console that oopses will be printed to.
> In a quick single-threaded benchmark, it looks like oopsing in a vfork()
> child with a very short stack trace only takes ~510 microseconds per run
> when a graphical console is active; but switching to a text console that
> oopses are printed to slows it down around 87x, to ~45 milliseconds per
> run.
> (Adding more threads makes this faster, but the actual oops printing
> happens under &die_lock on x86, so you can maybe speed this up by a factor
> of around 2 and then any further improvement gets eaten up by lock
> contention.)
> 
> It looks like it would take around 8-12 days to overflow a 32-bit counter
> with repeated oopsing on a multi-core X86 system running a graphical
> environment; both me (in an X86 VM) and Seth (with a distro kernel on
> normal hardware in a standard configuration) got numbers in that ballpark.
> 
> 12 days aren't *that* short on a desktop system, and you'd likely need much
> longer on a typical server system (assuming that people don't run graphical
> desktop environments on their servers), and this is a *very* noisy and
> violent approach to exploiting the kernel; and it also seems to take orders
> of magnitude longer on some machines, probably because stuff like EFI
> pstore will slow it down a ton if that's active.

I found a blog article[1] recommending LTS kernels to backport this as below.

    While this patch is already upstream, it is important that distributed
    kernels also inherit this oops limit and backport it to LTS releases if we
    want to avoid treating such null-dereference bugs as full-fledged security
    issues in the future.

Do you have a plan to backport this into upstream LTS kernels?

[1] https://googleprojectzero.blogspot.com/2023/01/exploiting-null-dereferences-in-linux.html


Thanks,
SJ

> 
> Signed-off-by: Jann Horn <jannh@...gle.com>
> Link: https://lore.kernel.org/r/20221107201317.324457-1-jannh@google.com
> Reviewed-by: Luis Chamberlain <mcgrof@...nel.org>
> Signed-off-by: Kees Cook <keescook@...omium.org>