linux-kernel - Re: [PATCH] gen_compile_command: Add support for separate KBUILD

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAD=FV=UHNwXhGcqua=gfGzyRMUt=fBd0RbRvQwttj-6hFdrjKw@mail.gmail.com>
Date:   Thu, 20 Jun 2019 13:57:00 -0700
From:   Doug Anderson <dianders@...gle.com>
To:     Nick Desaulniers <ndesaulniers@...gle.com>
Cc:     Matthias Kaehlcke <mka@...omium.org>,
        Tom Roeder <tmroeder@...gle.com>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Raul E Rangel <rrangel@...omium.org>,
        Tom Hughes <tomhughes@...omium.org>,
        Ryan Case <ryandcase@...omium.org>,
        Yu Liu <yudiliu@...gle.com>,
        Nathan Chancellor <natechancellor@...il.com>
Subject: Re: [PATCH] gen_compile_command: Add support for separate
 KBUILD_OUTPUT directory

Hi,

On Thu, Jun 20, 2019 at 1:25 PM Nick Desaulniers
<ndesaulniers@...gle.com> wrote:
>
> On Thu, Jun 20, 2019 at 1:13 PM Doug Anderson <dianders@...gle.com> wrote:
> > On Thu, Jun 20, 2019 at 12:53 PM Nick Desaulniers
> > <ndesaulniers@...gle.com> wrote:
> > >
> > > I do miss Doug's Kbuild caching patches' speedup.
> >
> > You actually get quite a bit of this by grabbing a new version of
> > ccache (assuming you use ccache).  :-P  You still have to pay the
> > penalty (twice) for all the options that are tested that the compiler
> > _doesn't_ support, but at least you get the cache for the commands
> > that the compiler does support.
>
> Hello darkness my old friend:
> https://nickdesaulniers.github.io/blog/2018/06/02/speeding-up-linux-kernel-builds-with-ccache/
> Man, that post has not aged well.  Here's what we do now:
> https://github.com/ClangBuiltLinux/continuous-integration/blob/45ab5842a69cb0c72d27d34e73b0599ec2a0e2ed/driver.sh#L227-L245
>
> > Specifically, make sure you have a ccache with:
> >
> >     * https://github.com/ccache/ccache/pull/365
> >     * https://github.com/ccache/ccache/pull/370
>
> Oh! Interesting finds and thanks for the pointers.  Did these make it
> into a release version of ccache, yet? If so, do you know which
> version?
>
> > I still have it in my thoughts to avoid the penalty for options that
> > the compiler doesn't support but haven't had time to work on it
> > recently.
>
> It had better not be autoconf! (Hopefully yet-to-be-written GNU C
> extensions can support feature detection via C preprocessor)

I've had a few ideas.  I won't object if you wanted to steal any of
them and implement them yourself.  :-P

1. Lamest but easiest (and best speedup for me personally) is to just
cheat and hack our toolchain wrapper (which is invoked _before_
ccache) to immediately reject flags we know our toolchain doesn't
support.  I don't love our toolchain wrapper since it adds ~15 ms to
every compiler invocation, but if it's there anyway might as well use
it.

2. Part of the slowness in testing for unsupported options is that
ccache runs twice.  After validating that it doesn't have a cache hit,
it first tries to produce a preprocessed version of the file.  With an
unsupported option that fails.  ...so ccache tries again _without_ the
preprocessor.  So you call the compiler twice.  We could either make
ccache skip this double-step when the target is /dev/null (why bother
trying to preprocess /dev/null?) or we could add a CCACHE directive
into the kernel build when testing options.  Probably the ccache
option makes the most sense.

3. In theory we could teach ccache how to cache these "tests for
unsupported options".  That might be hard, though.

-Doug