linux-hardening - Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAK7LNAQa_ZX=ULUgu+_YNe=A+4kgaZakzdM2Y6QPSWWfnEA2JQ@mail.gmail.com>
Date: Tue, 13 May 2025 23:52:46 +0900
From: Masahiro Yamada <masahiroy@...nel.org>
To: Kees Cook <kees@...nel.org>
Cc: Nathan Chancellor <nathan@...nel.org>, Nicolas Schier <nicolas.schier@...ux.dev>, 
	Petr Pavlu <petr.pavlu@...e.com>, Sebastian Andrzej Siewior <bigeasy@...utronix.de>, 
	Justin Stitt <justinstitt@...gle.com>, Marco Elver <elver@...gle.com>, 
	Andrey Konovalov <andreyknvl@...il.com>, Andrey Ryabinin <ryabinin.a.a@...il.com>, 
	Richard Weinberger <richard@....at>, Anton Ivanov <anton.ivanov@...bridgegreys.com>, 
	Johannes Berg <johannes@...solutions.net>, linux-kernel@...r.kernel.org, 
	linux-hardening@...r.kernel.org, linux-kbuild@...r.kernel.org, 
	kasan-dev@...glegroups.com, linux-um@...ts.infradead.org
Subject: Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

On Fri, May 9, 2025 at 8:59 AM Kees Cook <kees@...nel.org> wrote:
>
> On Fri, May 09, 2025 at 08:13:18AM +0900, Masahiro Yamada wrote:
> > On Fri, May 9, 2025 at 1:56 AM Kees Cook <kees@...nel.org> wrote:
> > >
> > > On Fri, May 09, 2025 at 01:44:09AM +0900, Masahiro Yamada wrote:
> > > > On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@...nel.org> wrote:
> > > > >
> > > > > On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > > > > > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@...nel.org> wrote:
> > > > > > >
> > > > > > >  v2:
> > > > > > >   - switch from -include to -I with a -D gated include compiler-version.h
> > > > > > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> > > > > >
> > > > > >
> > > > > > What do you think of my patch as a prerequisite?
> > > > > > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > > > > > Perhaps, can you implement this series more simply?
> > > > > >
> > > > > > My idea is to touch a single include/generated/global-rebuild.h
> > > > > > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> > > > > >
> > > > > > When the file is touched, the entire kernel source tree will be rebuilt.
> > > > > > This may rebuild more than needed (e.g. vdso) but I do not think
> > > > > > it is a big deal.
> > > > >
> > > > > This is roughly where I started when trying to implement this, but I
> > > > > didn't like the ergonomics of needing to scatter "touch" calls all over,
> > > > > which was especially difficult for targets that shared a build rule but
> > > > > may not all need to trigger a global rebuild. But what ultimately pushed
> > > > > me away from it was when I needed to notice if a non-built source file
> > > > > changed (the Clang .scl file), and I saw that I need to be dependency
> > > > > driven rather than target driven. (Though perhaps there is a way to
> > > > > address this with your global-rebuild.h?)
> > > > >
> > > > > As far as doing a full rebuild, if it had been available last week, I
> > > > > probably would have used it, but now given the work that Nicolas, you,
> > > > > and I have put into this, we have a viable way (I think) to make this
> > > > > more specific. It does end up being a waste of time/resources to rebuild
> > > > > stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> > > > > does add up when I'm iterating on something that keeps triggering a full
> > > > > rebuild. We already have to do the argument filtering for targets that
> > > > > don't want randstruct, etc, so why not capitalize on that and make the
> > > > > rebuild avoid those files too?
> > > >
> > > >
> > > > efi-stub, vdso are very small.
> > > >
> > > > Unless this turns out to be painful, I prefer
> > > > a simpler implementation.
> > > >
> > > > You will see how .scl file is handled.
> > > >
> > > > See the below code:
> > > >
> > > >
> > > > diff --git a/Kbuild b/Kbuild
> > > > index f327ca86990c..85747239314c 100644
> > > > --- a/Kbuild
> > > > +++ b/Kbuild
> > > > @@ -67,10 +67,20 @@ targets += $(atomic-checks)
> > > >  $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
> > > >         $(call if_changed,check_sha1)
> > > >
> > > > +rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
> > > > scripts/gcc-plugins/, $(GCC_PLUGIN))
> > > > +rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h
> > >
> > > These are in $(objtree)
> >
> > Yes.
> >
> > > > +rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl
> > >
> > > This is in $(srctree)
> >
> > Yes.
> >
> > > > +
> > > > +quiet_cmd_touch = TOUCH   $@
> > > > +      cmd_touch = touch $@
> > > > +
> > > > +include/generated/global-rebuild.h: $(rebuild-y)
> > > > +       $(call cmd,touch)
> > >
> > > Is this rule going to find the right versions of the dependencies?
> >
> > I think so, but please test it.
>
> The patch was white-space damaged and wrapped, but I rebuilt it manually
> and it mostly works. There still seems to be some ordering issues, as
> some stuff gets rebuilt on a record build:
>
> # Clean the tree and pick an "everything" build
> $ make O=gcc-test clean allmodconfig -s
>
> # Make a target normally
> $ make O=gcc-test kernel/seccomp.o -s
>
> # Touch a gcc plugin that was in .config
> $ touch scripts/gcc-plugins/stackleak_plugin.c
>
> # Build and a full rebuild is triggered (good)
> $ make O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   HOSTCXX scripts/gcc-plugins/stackleak_plugin.so
>   INSTALL libsubcmd_headers
>   TOUCH   include/generated/global-rebuild.h
>   CC      kernel/bounds.s
>   CC      arch/x86/kernel/asm-offsets.s
>   CALL    ../scripts/checksyscalls.sh
>   CC      kernel/seccomp.o
> make[1]: Leaving directory '/srv/code/gcc-test'
>
> # Build again, but more stuff gets built
> $ make O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   CC      scripts/mod/empty.o
>   CC      scripts/mod/devicetable-offsets.s
>   INSTALL libsubcmd_headers
>   MKELF   scripts/mod/elfconfig.h
>   HOSTCC  scripts/mod/modpost.o
>   HOSTCC  scripts/mod/sumversion.o
>   HOSTCC  scripts/mod/symsearch.o
>   HOSTCC  scripts/mod/file2alias.o
>   HOSTLD  scripts/mod/modpost
>   CALL    ../scripts/checksyscalls.sh
> make[1]: Leaving directory '/srv/code/gcc-test'
>
> # Third time finally everything is stable
> $ hmake O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   CALL    ../scripts/checksyscalls.sh
>   INSTALL libsubcmd_headers
> make[1]: Leaving directory '/srv/code/gcc-test'
>
>
> Note that scripts/mod/* gets rebuilt on the second rebuild.

Hmm.
OK, my code did not work.

I accept your patch set (although I am not a big fan
of the added complexity...)

Could you move the normalize_path macro
to scripts/Kbuild?


-- 
Best Regards
Masahiro Yamada