linux-kernel - Re: [v2 2/2] kbuild: copy scripts/atomic/atomic-*.h to include/generated/atomic-*.h

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK7LNASqZxn3e-DSO0cnYRZ7nY-ztt7rLUry9Fqez+ZT5VMeQw@mail.gmail.com>
Date:   Mon, 16 May 2022 18:30:41 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     Mark Rutland <mark.rutland@....com>
Cc:     Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Vincent Mailhol <mailhol.vincent@...adoo.fr>,
        Will Deacon <will@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>
Subject: Re: [v2 2/2] kbuild: copy scripts/atomic/atomic-*.h to include/generated/atomic-*.h

On Mon, May 16, 2022 at 5:43 PM Mark Rutland <mark.rutland@....com> wrote:
>
> [adding the atomics maintainers to Cc]
>
> On Sat, May 14, 2022 at 01:02:07PM +0900, Masahiro Yamada wrote:
> > include/linux/atomic/*.h are generated by scripts/atomic/gen-atomics.sh.
> >
> > To ensure they are not edited, scripts/atomic/check-atomics.sh checks
> > sha1sum every time.
> >
> > This commit moves include/linux/atomic/*.h to scripts/atomic/*.h_shipped,
> > which are copied to include/generated/ at build time:
> >
> >   COPY    include/generated/atomic-instrumented.h
> >   COPY    include/generated/atomic-long.h
> >   COPY    include/generated/atomic-arch-fallback.h
>
> FWIW, moving these and copying them at build time is fine by me, given that
> better indicates that these are generated.
>
> > I dropped the sha1sum checks. I hope nobody would try to directly modify
> > *_shipped files.
>
> I'd prefer to keep the sha1sum check, because we're worried that people *will*
> modify them, and that won't be noticed until the next time they're regenerated.

OK, but is there any reason to embed the checksum to the headers?

Generally, you can have *.sha1 file to collect the checksums,
and use the "sha1sum --check" command.

Maybe, scripts/atomic/check-atomics.sh was unneeded.


$ sha1sum include/linux/atomic/*.h   >  scripts/atomic/atomics.sha1
$ cat scripts/atomic/atomics.sha1
c0f1a9e951f38ccfa146ca2431f9e1611191a402
include/linux/atomic/atomic-arch-fallback.h
97ce73d2c176725d199a810eb81c574022ffa899
include/linux/atomic/atomic-instrumented.h
b0a5ee2e9497a41795644fa115df184d6331b9c2  include/linux/atomic/atomic-long.h
$ sha1sum --check   scripts/atomic/atomics.sha1
include/linux/atomic/atomic-arch-fallback.h: OK
include/linux/atomic/atomic-instrumented.h: OK
include/linux/atomic/atomic-long.h: OK


It is possible to do "sha1sum --check && cp".



> > Kbuild runs more and more tools at build time these days because they
> > are fast enough on modern systems.
> >
> > For example,
> >
> >  - 29c833061c1d8c2d1d23a62e7061561eadd76cdb
> >    ("kconfig: generate lexer and parser during build instead of shipping")
> >
> >  - 7c0303ff7e67b637c47d8afee533ca9e2a02359b
> >    ("crypto: arm - generate *.S by Perl at build time instead of shipping them")
> >
> > Yet, gen-atomics.sh is too slow.
>
> Yes; we'd originally wanted to run them at build time, but that was too slow,
> and as a compromise we moved to regenerating them whenever we changed the
> scripting.

I remember it.

https://lore.kernel.org/lkml/20181123153321.8561-1-mark.rutland@arm.com/


>
> > I guess it can be improved because it is just simple text processing.
> > Then, Kbuild can execute it at build time.
>
> It is in theory possible to make them much faster, yes. The actual slowness of
> the scripting is largely due to using sub-shells resulting in a load of
> fork+exec, which could be avoided if using a real scripting language.
>
> Practically speaking though, we're rather limited in what we can rely upon
> being available. We chose to use POSIX shell as a lowest-common-demoninator,
> and anything that'd be *nice* to use isn't going to always be available,
> meaning that even if we make it faster we can't necessarily build it all the
> time anyway.


Kernel builds already rely on Perl.

The base building of the kernel does not depend on Python,
but some targets such as "make clang-tidy", "make clang-analyzer"
need Python3.


>
> Thanks,
> Mark.


BTW, full-quoting to this thread is not a good idea.
I cut down the code diff.




-- 
Best Regards
Masahiro Yamada