linux-kernel - Re: [tip:locking/core] locking/atomics: Check generated headers are up-to-date

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20181121114028.iiuleuwuayednvut@salmiak>
Date:   Wed, 21 Nov 2018 11:40:29 +0000
From:   Mark Rutland <mark.rutland@....com>
To:     Ingo Molnar <mingo@...nel.org>
Cc:     tglx@...utronix.de, hpa@...or.com, linux-kernel@...r.kernel.org,
        peterz@...radead.org, boqun.feng@...il.com, will.deacon@....com,
        linux-tip-commits@...r.kernel.org, akpm@...ux-foundation.org,
        torvalds@...ux-foundation.org
Subject: Re: [tip:locking/core] locking/atomics: Check generated headers are
 up-to-date

Hi Ingo,

On Wed, Nov 21, 2018 at 09:02:58AM +0100, Ingo Molnar wrote:
> 
> * tip-bot for Mark Rutland <tipbot@...or.com> wrote:
> 
> > Commit-ID:  8d32588077bdc390420cfa6946f407033a20d7a8
> > Gitweb:     https://git.kernel.org/tip/8d32588077bdc390420cfa6946f407033a20d7a8
> > Author:     Mark Rutland <mark.rutland@....com>
> > AuthorDate: Tue, 4 Sep 2018 11:48:29 +0100
> > Committer:  Ingo Molnar <mingo@...nel.org>
> > CommitDate: Thu, 1 Nov 2018 11:01:10 +0100
> > 
> > locking/atomics: Check generated headers are up-to-date
> > 
> > Now that all the generated atomic headers are in place, it would be good
> > to ensure that:
> > 
> > a) the headers are up-to-date when scripting changes.
> > 
> > b) developers don't directly modify the generated headers.
> > 
> > To ensure both of these properties, let's add a Kbuild step to check
> > that the generated headers are up-to-date.
> > 
> > Signed-off-by: Mark Rutland <mark.rutland@....com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
> > Cc: linux-arm-kernel@...ts.infradead.org
> > Cc: catalin.marinas@....com
> > Cc: Will Deacon <will.deacon@....com>
> > Cc: linuxdrivers@...otech.com
> > Cc: dvyukov@...gle.com
> > Cc: Boqun Feng <boqun.feng@...il.com>
> > Cc: arnd@...db.de
> > Cc: aryabinin@...tuozzo.com
> > Cc: glider@...gle.com
> > Link: http://lkml.kernel.org/r/20180904104830.2975-6-mark.rutland@arm.com
> > Signed-off-by: Ingo Molnar <mingo@...nel.org>
> > ---
> >  Kbuild                          | 18 ++++++++++++++++--
> >  scripts/atomic/check-atomics.sh | 19 +++++++++++++++++++
> >  2 files changed, 35 insertions(+), 2 deletions(-)
> 
> These scripts are *awfully* slow to be run at every kernel build - even a 
> reasonably fast machine:

Andrew also reported this, and I'm trying to figure out what to do.

I looked into optimizing the scripts, but that turned out to be a dead end.
The majority of the cost is in the fork() etc for sub-shells, which the scripts
make heavy use of to capture string return values. Trying to avoid sub-shells
rendered the scripts unreadable.

> Could we please get this fixed so that proper dependencies are checked 
> and it's only regenerated when needed? This slowdown makes additive-build 
> kernel development quite painful, as ~5 seconds is in the 'too long' 
> category already, while 1.2 seconds is basically instantaneous.

Just to check, are we happy to eat the full cost for the first build of a
pristine tree?

One reason we do the check rather than (re-)generating the headers is that
Linus requested [1] the generated header be committed so that they show up in
git grep, but it looks like he was happy to be flexible on that.

If we're happy to not commit in the generated headers, and if we're happy to
pay the cost for a pristine tree, that's fairly straightforward to do.
Otherwise, this has to be an optional check.

Thoughts?

Thanks,
Mark.

[1] https://lkml.kernel.org/r/CA+55aFxjU0op8QLLu0n-RjHBs7gQsLvD8jcyedgw6jeZFN7B+Q@mail.gmail.com