linux-kernel - [PATCH] tools build: Fix parallel perf build when building in-tree with O=

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-Id: <20220307140047.26910-1-thomas.de_schampheleire@nokia.com>
Date:   Mon,  7 Mar 2022 15:00:47 +0100
From:   Thomas De Schampheleire <thomas.de_schampheleire@...ia.com>
To:     linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Arnaldo Carvalho de Melo <acme@...nel.org>,
        Mark Rutland <mark.rutland@....com>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Jiri Olsa <jolsa@...nel.org>,
        Namhyung Kim <namhyung@...nel.org>,
        Mathias De Maré <mathias.de_mare@...ia.com>,
        Joeri Barbarien <joeri.barbarien@...ia.com>,
        Thomas De Schampheleire <thomas.de_schampheleire@...ia.com>
Subject: [PATCH] tools build: Fix parallel perf build when building in-tree with O=

From: Mathias De Maré <mathias.de_mare@...ia.com>

Parallel compilation of perf using following command may fail:
    make C=tools/perf O=tools/perf JOBS=81
Such command is executed from the Buildroot embedded Linux build system [1]

The encountered error is:

  ld: .../tools/perf/perf-in.o: in function `parse_events__scanner':
  .../tools/perf/util/parse-events.c:2204: undefined reference to `parse_events_parse'
  ld: .../tools/perf/util/parse-events.c:2204: undefined reference to `parse_events_parse'
  ld: .../tools/perf/util/parse-events.c:2204: undefined reference to `parse_events_parse'
  collect2: error: ld returned 1 exit status
  make[2]: *** [Makefile.perf:677: .../tools/perf/perf] Error 1
  make[1]: *** [Makefile.perf:240: sub-make] Error 2
  make: *** [Makefile:70: all] Error 2
  make: Leaving directory '.../tools/perf'

The problem only occurs when 'O=tools/perf' (i.e. output directory is same
as source directory) and not without O= or with a different output
directory.

This problem happens when parse-events-bison.o is compiled while
parse-events-bison.c is still being written. This is a dependency problem,
but a subtle one.

File tools/build/Makefile.build has a rule with dependency:
  $(OUTPUT)%.o: %.c

For the generation of the .c file, tools/perf/util/Build has a rule:
  $(OUTPUT)util/parse-events-bison.c $(OUTPUT)util/parse-events-bison.h: util/parse-events.y

The clue is that the .o rule depends on %.c without $(OUTPUT) prefix, while
the .c rule has $(OUTPUT) included.

If OUTPUT is 'tools/perf', then make sees:
  .../tools/perf/util/parse-events-bison.o: util/parse-events-bison.c
and
  .../tools/perf/util/parse-events-bison.c: ...
and it does not consider both .c files to be the same when calculating
dependencies.
As a result, the .o rule is allowed to start as soon as the .c file is
found, without the corresponding rule necessarily already having completed.

The problem can easily be reproduced with following change:

: diff --git a/tools/perf/util/Build b/tools/perf/util/Build
: index 2e5bfbb69960..447636f6db7f 100644
: --- a/tools/perf/util/Build
: +++ b/tools/perf/util/Build
: @@ -221,6 +221,8 @@ $(OUTPUT)util/parse-events-flex.c $(OUTPUT)util/parse-events-flex.h: util/parse-

:  $(OUTPUT)util/parse-events-bison.c $(OUTPUT)util/parse-events-bison.h: util/parse-events.y
:  	$(call rule_mkdir)
: +	touch $@
: +	sleep 60
:  	$(Q)$(call echo-cmd,bison)$(BISON) -v $< -d $(PARSER_DEBUG_BISON) $(BISON_FILE_PREFIX_MAP) \
:  		-o $(OUTPUT)util/parse-events-bison.c -p parse_events_

This change enforces that the parse-events-bison.c file _exists_, but is not
yet valid.

The debug output of 'make -d' confirms what happens:

    [...]
        Considering target file '.../tools/perf/util/parse-events-bison.c'.
         File '.../tools/perf/util/parse-events-bison.c' does not exist.
         Finished prerequisites of target file '.../tools/perf/util/parse-events-bison.c'.
        Must remake target '.../tools/perf/util/parse-events-bison.c'.
touch .../tools/perf/util/parse-events-bison.c
        Recipe of '.../tools/perf/util/parse-events-bison.c' is being run.
      Pruning file '.../tools/perf/util/parse-events-bison.c'.
    [...]
    Considering target file '.../tools/perf/util/parse-events-bison.o'.
     File '.../tools/perf/util/parse-events-bison.o' does not exist.
     Looking for an implicit rule for '.../tools/perf/util/parse-events-bison.o'.
     Trying implicit prerequisite 'util/parse-events-bison.c'.
     Found prerequisite 'util/parse-events-bison.c' as VPATH '.../tools/perf/util/parse-events-bison.c'
     Found an implicit rule for '.../tools/perf/util/parse-events-bison.o'.
      Considering target file 'util/parse-events-bison.c'.
       Looking for an implicit rule for 'util/parse-events-bison.c'.
       No implicit rule found for 'util/parse-events-bison.c'.
       Finished prerequisites of target file 'util/parse-events-bison.c'.
      No need to remake target 'util/parse-events-bison.c'.
     Finished prerequisites of target file '.../tools/perf/util/parse-events-bison.o'.
    Must remake target '.../tools/perf/util/parse-events-bison.o'.
    Recipe of '.../tools/perf/util/parse-events-bison.o' is being run.
    [...]

The rule for parse-events-bison.o considers the target
'util/parse-events-bison.c', finds that it exists (thanks to the touch
reproduction) and then starts the .o generation, which will compile an
empty/incomplete file.

The problem can be fixed by adding an additional rule in
tools/build/Makefile.build, to enforce the dependency of
parse-events-bison.o on $(OUTPUT)/.../parse-events-bison.c _including_ the
$(OUTPUT) prefix. This results in:

  [...]
  Considering target file '.../tools/perf/util/parse-events-bison.o'.
   File '.../tools/perf/util/parse-events-bison.o' does not exist.
   Looking for an implicit rule for '.../tools/perf/util/parse-events-bison.o'.
   Trying implicit prerequisite '.../tools/perf/util/parse-events-bison.c'.
   Found an implicit rule for '.../tools/perf/util/parse-events-bison.o'.
    Pruning file '.../tools/perf/util/parse-events-bison.c'.
   Finished prerequisites of target file '.../tools/perf/util/parse-events-bison.o'.
  The prerequisites of '.../tools/perf/util/parse-events-bison.o' are being made.
  [...]

and here make understood that the .c file is still being generated and waits
on its completion.

[1] https://git.buildroot.net/buildroot/tree/package/linux-tools/linux-tool-perf.mk.in#n158

Co-developed-by: Joeri Barbarien <joeri.barbarien@...ia.com>
Signed-off-by: Joeri Barbarien <joeri.barbarien@...ia.com>
Signed-off-by: Mathias De Maré <mathias.de_mare@...ia.com>
Co-developed-by: Thomas De Schampheleire <thomas.de_schampheleire@...ia.com>
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@...ia.com>
---
 tools/build/Makefile.build | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/build/Makefile.build b/tools/build/Makefile.build
index 715092fc6a23..0f9ec51d9c2e 100644
--- a/tools/build/Makefile.build
+++ b/tools/build/Makefile.build
@@ -92,6 +92,10 @@ ifneq ($(filter $(obj),$(hostprogs)),)
 endif
 
 # Build rules
+$(OUTPUT)%-bison.o: $(OUTPUT)%-bison.c FORCE
+	$(call rule_mkdir)
+	$(call if_changed_dep,$(host)cc_o_c)
+
 $(OUTPUT)%.o: %.c FORCE
 	$(call rule_mkdir)
 	$(call if_changed_dep,$(host)cc_o_c)
-- 
2.34.1