Date:   Wed, 23 Jan 2019 21:32:16 -0500
From:   Joel Fernandes <joel@...lfernandes.org>
To:     Daniel Colascione <dancol@...gle.com>
Cc:     Karim Yaghmour <karim.yaghmour@...rsys.com>,
        "H. Peter Anvin" <hpa@...or.com>,
        Greg KH <gregkh@...uxfoundation.org>,
        Christoph Hellwig <hch@...radead.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>, ast@...nel.org,
        atish patra <atishp04@...il.com>,
        Borislav Petkov <bp@...en8.de>, Ingo Molnar <mingo@...hat.com>,
        Jan Kara <jack@...e.cz>, Jonathan Corbet <corbet@....net>,
        Kees Cook <keescook@...omium.org>, kernel-team@...roid.com,
        "open list:DOCUMENTATION" <linux-doc@...r.kernel.org>,
        Manoj Rao <linux@...ojrajarao.com>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Paul McKenney <paulmck@...ux.vnet.ibm.com>,
        "Peter Zijlstra (Intel)" <peterz@...radead.org>,
        Randy Dunlap <rdunlap@...radead.org>, rostedt@...dmis.org,
        Thomas Gleixner <tglx@...utronix.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@...nel.org>,
        yhs@...com
Subject: Re: [RFC] Provide in-kernel headers for making it easy to extend the
 kernel

On Wed, Jan 23, 2019 at 02:37:47PM -0800, Daniel Colascione wrote:
> On Wed, Jan 23, 2019 at 1:29 PM Karim Yaghmour
> <karim.yaghmour@...rsys.com> wrote:
[...]
> > Personally I advocated a more aggressive approach with Joel in private:
> > just put the darn headers straight into the kernel image, it's the
> > *only* artifact we're sure will follow the Android device whatever
> > happens to it (like built-in ftrace).
> 
> I was thinking along similar lines. Ordinarily, we make loadable
> kernel modules. What we kind of want here is a non-loadable kernel
> module --- or a non-loadable section in the kernel image proper. I'm
> not familiar with early-stage kernel loader operation: I know it's
> possible to crease discardable sections in the kernel image, but can
> we create sections that are never slurped into memory in the first
> place? If not, maybe loading and immediately discarding the header
> section is good enough.

I am happy to see if I can shrink it down further. Especially using xz and
stripping all comments period. I am optimistic this can be brought down
further to a point where it would make sense to everyone to build it into the
kernel. Lets see.

Last time I stripped comments, it went down by ~40%. What I haven't tried is
doing this *with* xz compression. I am also open to brainstorming what else
can be stripped.

OTOH the reason I didn't focus much on size is: modules are pretty much
universal and I'm confident of wide spread use of this feature for Android-based
products and not needing of "chasing headers" if we modularize it, since
Android's project treble has modularized things and modules are now default
enabled. Putting headers into a module lets us enjoy the ride there.

I am quite hosed for the next week or so to work on this, but I should be
able to get back to it after.

cheers,

 - Joel


> 
> > To that end, I even had some crazy
> > ideas on how to compress the headers even further than with std
> > compression algorithms -- here's a snippet from an email I sent Joel
> > some time back detailing such a hack:
> > > Since C headers have fairly constrained semantics and since the types of semantics generally used to name structs, etc. in the Linux kernel are well established, we can likely devise a very customized compression algorithm for the purpose.
> 
> Would such a thing really do better than LZMA? LZMA already has very
> clever techniques for eliminating long-range redundancies in
> compressible text, including redundancies at the sub-byte level. I can
> certainly understand the benefit of stripping comments, since removing
> comments really does decrease the total amount of information the
> compressor has to preserve, but I'm not sure how much the encoding
> scheme you propose below would help, since it reminds me of the
> encoding scheme that LZMA would discover automatically.
> 
> > Whether such craziness makes sense or is adopted or not isn't mine to
> > chart, but I certainly can't see eBPF reaching the same mass deployment
> > ftrace has within the Android ecosystem until there's a way to use it
> > without having to chase kernel headers independently of kernel images.
> > There are "too many clicks" involved and someone somewhere will drop the
> > ball if it's not glued to the kernel in some way shape or form. Any
> > solution that solves this is one I'd love to hear about.
> 
> I agree. There definitely needs to be a "just collect a damn trace"
> button that works on any device, and for this button to work and
> incorporate eBPF, the system needs to be able to describe itself.

Powered by blists - more mailing lists