Message-Id: <20190308212524.121126-1-dianders@chromium.org>
Date:   Fri,  8 Mar 2019 13:25:23 -0800
From:   Douglas Anderson <dianders@...omium.org>
To:     Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Michal Marek <michal.lkml@...kovi.net>
Cc:     groeck@...omium.org, briannorris@...omium.org, swboyd@...omium.org,
        Douglas Anderson <dianders@...omium.org>,
        linux-kernel@...r.kernel.org, linux-kbuild@...r.kernel.org
Subject: [PATCH] kbuild: Speed up install, modules_install and kernelrelease

As per the description of the old commit 3298b690b21c ("kbuild: Add a
cache for generated variables"), calling the C compiler lots of times
during the parsing stage of the Makefile can be a little slow.  If you
happen to have a C compiler whose fork/exec time isn't optimized
(perhaps it was linked dynamically, or perhaps it's called through
several wrappers, or ...) then this can be more than a little slow, it
can be very slow.

The above slowness was addressed with the old Makefile cache, but the
cache was reverted in commit e08d6de4e532 ("kbuild: remove kbuild
cache").  That commit indicated that we expected to get the speed
gains back because "the result of compiler tests will be naturally
cached in the .config file", but as of the latest mainline there are
still lots of direct calls to the C compiler that happen in the
Makefile parsing stage.

Perhaps we could try re-introducing the Makefile cache and fix the
problems it was causing, but this patch codes up another solution that
gets some of the speed gains back, perhaps with less complexity.

Specifically I observed that by timing the parsing stage of the
Makefile by adding statements like this to the beginning and end of
the Makefile:
  ZZZ := $(shell echo "$$(date '+%T.%N'): ..." >> /tmp/timing.txt)

...that three targets were spending 3 seconds each parsing the
Makefile in my environment (building a Chrome OS kernel with clang)
when it felt like they ought to be quick.  These were: install,
modules_install and kernelrelease

I saw timings that looked like:
  09:23:08.903204451: Parsing Makefile, goals: install
  09:23:11.960515772: Done parsing Makefile, goals: install

Given that the kbuild system doesn't sync the config for any of these
options I believe it is safe to say that it can't compile any code.
Thus if we can avoid the C compiler invocation here we'll save time.

The simplest way to accomplish this is to neuter all the calls to the
compiler by stubbing them out.  After doing so (AKA applying this
change) the same timing looks like:
  09:28:04.929252271: Parsing Makefile, goals: install
  09:28:05.107468449: Done parsing Makefile, goals: install

During an incremental kernel build/install on Chrome OS we call
install once, modules_install twice (once to keep unstripped), and
kernelrelease once.  Thus this saves ~12 seconds on an incremental
kernel build/install.  With my current setup, this brings it from ~40
seconds to ~28 seconds.  Re-introducing the Makefile cache would
likely save us an extra ~3 seconds since we'd avoid the parsing in the
compile stage.

Signed-off-by: Douglas Anderson <dianders@...omium.org>
---

 Makefile | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/Makefile b/Makefile
index f070e0d65186..5f7532d9be65 100644
--- a/Makefile
+++ b/Makefile
@@ -304,6 +304,20 @@ else
 scripts/Kbuild.include: ;
 include scripts/Kbuild.include
 
+# If we can't sync the config then we know we can't compile code.
+# Replace the slow option-testing commands with stubs to significantly
+# speed up the Makefile parsing stage of the build.
+ifeq ($(may-sync-config),0)
+	ar-option :=
+	as-option :=
+	cc-disable-warning :=
+	cc-ifversion :=
+	cc-ldoption :=
+	cc-option :=
+	cc-option-yn := n
+	ld-option :=
+endif
+
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-- 
2.21.0.360.g471c308f928-goog

Powered by blists - more mailing lists