linux-kernel - Re: [PATCHv2] locking/atomics: build atomic headers as required

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 6 Dec 2018 17:45:16 +0000
From:   Mark Rutland <mark.rutland@....com>
To:     Ingo Molnar <mingo@...nel.org>
Cc:     linux-kernel@...r.kernel.org,
        Andrew Morton <akpm@...ux-foundation.org>,
        Boqun Feng <boqun.feng@...il.com>,
        Borislav Petkov <bp@...e.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Will Deacon <will.deacon@....com>
Subject: Re: [PATCHv2] locking/atomics: build atomic headers as required

On Thu, Dec 06, 2018 at 05:13:44PM +0100, Ingo Molnar wrote:
> 
> * Mark Rutland <mark.rutland@....com> wrote:
> 
> > Andrew and Ingo report that the check-atomics.sh script is simply too
> > slow to run for every kernel build, and it's impractical to make it
> > faster without rewriting it in something other than shell.
> > 
> > Rather than committing the generated headers, let's regenerate these
> > as-required, if we change the data or scripts used to generate the
> > atomic headers, or when building from a pristine tree.
> > 
> > That ensures they're always up-to-date, allows them to be built in
> > parallel, and avoid redundant rebuilds, which is a 2-8s saving per
> > incremental build. Since the results are not committed, it's very
> > obvious that they should not be modified directly. If we need to
> > generate more headers in future, it's easy to extend Makefile.genheader
> > to permit this.
> > 
> > I've verified that this works in the cases we previously had issues with
> > (out-of-tree builds and where scripts have no execute permissions), and
> > have tested these cases for both x86_64 and arm64.
> > 
> > The diffstat looks nice, at least...
> > 
> > Signed-off-by: Mark Rutland <mark.rutland@....com>
> > Cc: Andrew Morton <akpm@...ux-foundation.org>
> > Cc: Boqun Feng <boqun.feng@...il.com>
> > Cc: Borislav Petkov <bp@...e.de>
> > Cc: Ingo Molnar <mingo@...nel.org>
> > Cc: Peter Zijlstra <peterz@...radead.org>
> > Cc: Will Deacon <will.deacon@....com>
> > Cc: linux-kernel@...r.kernel.org
> > ---
> >  Kbuild                                    |   18 +-
> >  Makefile                                  |    8 +-
> >  arch/arm64/include/asm/atomic.h           |    2 +-
> >  arch/x86/include/asm/atomic.h             |    2 +-
> >  include/asm-generic/atomic-instrumented.h | 1787 ----------------------
> >  include/asm-generic/atomic-long.h         | 1012 -------------
> >  include/linux/atomic-fallback.h           | 2294 -----------------------------
> >  include/linux/atomic.h                    |    4 +-
> >  scripts/Makefile.genheader                |   26 +
> >  scripts/atomic/check-atomics.sh           |   19 -
> >  10 files changed, 38 insertions(+), 5134 deletions(-)
> >  delete mode 100644 include/asm-generic/atomic-instrumented.h
> >  delete mode 100644 include/asm-generic/atomic-long.h
> >  delete mode 100644 include/linux/atomic-fallback.h
> >  create mode 100644 scripts/Makefile.genheader
> >  delete mode 100755 scripts/atomic/check-atomics.sh
> 
> So these 'automatically generated' headers are actual and important code, 
> and I think it's bad practice to remove these from the git grep search 
> space and history as well.

Ok. Let's aim for that, then. :)

> Really, this whole notion of auto-generating the headers should be 
> implemented correctly, instead of working around deficiencies in a 
> short-term fashion that introduces other deficiencies.

I certainly agree.

> I never got any replies to my previous comments about this:
> 
>   <20181128083057.GA7879@...il.com>
> 
> Did I miss some mails of yours perhaps?

It appears my mailserver dropped that on the floor, as I can't find it
in any of my mailboxes. :(

Sorry about that; I'll try to figure out what happened there.

This patch covers your suggestion of parallelizing things. On my box
with -j64, that saves ~1.5s for a pristine allnoconfig build which
otherwise takes ~15s.

Practically, optimizing the scripts requires moving away from shell.
The cost is all down to sub-shells resulting in fork+exec, but the
scripts would be illegible without this.

One option is to generate a checksum when the headers are generated and
checked in, and check that at build time. I'll look into this tomorrow.

Thanks,
Mark.