linux-kernel - Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20201015075202.GD13775@alley>
Date:   Thu, 15 Oct 2020 09:52:02 +0200
From:   Petr Mladek <mladek.petr@...il.com>
To:     John Ogness <john.ogness@...utronix.de>
Cc:     Geert Uytterhoeven <geert@...ux-m68k.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Rasmus Villemoes <linux@...musvillemoes.dk>,
        Peter Zijlstra <peterz@...radead.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Daniel Thompson <daniel.thompson@...aro.org>
Subject: Re: [GIT PULL] printk for 5.10 (includes lockless ringbuffer)

On Wed 2020-10-14 23:01:30, John Ogness wrote:
> On 2020-10-14, Geert Uytterhoeven <geert@...ux-m68k.org> wrote:
> >> - Fully lockless ringbuffer implementation, including the support for
> >>   continuous lines. It will allow to store and read messages in any
> >>   situation wihtout the risk of deadlocks and without the need
> >>   of temporary per-CPU buffers.
> >
> >     linux-m68k-atari_defconfig$ bloat-o-meter vmlinux.old vmlinux.lockless_ringbuffer
> >     add/remove: 39/16 grow/shrink: 9/15 up/down: 214075/-4362 (209713)
> >     Function                                     old     new   delta
> >     _printk_rb_static_infos                        -  180224 +180224
> >     _printk_rb_static_descs                        -   24576  +24576
> 
> 131072 of _printk_rb_static_infos is reserved for dictionary usage. The
> rest (49152) is for the record meta data. Previously any dictionary
> usage and record meta data was embedded in the message buffer (log_buf,
> 65536).
> 
> Since the meta data is now separate, setting CONFIG_LOG_BUF_SHIFT=15
> would provide roughly the same amount of record storage as
> CONFIG_LOG_BUF_SHIFT=16 did with vmlinux.old. Then there would be:
> 
> 32768: message buffer
> 24576: meta data
> 65536: dictionary data
> 12288: descriptor data
> 
> Excluding the dictionary data, the total is 65536.
> 
> (With vmlinux.old there is 65536 total with CONFIG_LOG_BUF_SHIFT=16.)
>
> It is the reserved dictionary data that is hurting us here.
> 
> Should we provide a config option to kill the dictionary data?

This might be the best solution. AFAIK, they are currently
used only by journalctl. IMHO, most people are not aware of it
and it is not a widely used feature.


> Should CONFIG_LOG_BUF_SHIFT do an implicit -1 so that the sizes (without
> dictionary data) are about the same as before?

Beware that this might result in slightly reduced amount of stored
messages. The storage of the dictionary meta data is less effective
because they are not longer stored together with the messages and
we need to guarantee a space for them.

I would prefer to keep the current global default. In average,
the size of the systems is growing. They produce more messages
and have more memory available.

Yeah, the space hurts on small systems. We might modify the default
for them. Or we could allow to disable the dictionary.

Note that this is not the full picture of the printk memory usage.
I believe that we will be able to get rid of printk_safe and printk_nmi.
It would safe 16kB per-CPU at runtime.

If we make the dictionary optional and remove the per-CPU buffers
then we might end-up with less memory needs than before.

Best Regards,
Petr