linux-kernel - Re: [RFC PATCH] kbuild: Add option to generate a Compilation Database

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAK7LNASnG26_McrN8w3wHbzVU1+rHsKwY4Gu4xp7ff8wyCdu+g@mail.gmail.com>
Date:   Wed, 19 Jun 2019 00:16:08 +0900
From:   Masahiro Yamada <yamada.masahiro@...ionext.com>
To:     Raul E Rangel <rrangel@...omium.org>
Cc:     Matthias Kaehlcke <mka@...omium.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        zwisler@...omium.org, Joe Lawrence <joe.lawrence@...hat.com>,
        Kees Cook <keescook@...omium.org>,
        Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Petr Mladek <pmladek@...e.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Michal Marek <michal.lkml@...kovi.net>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Changbin Du <changbin.du@...el.com>,
        Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>,
        Sri Krishna chowdary <schowdary@...dia.com>,
        Matthew Wilcox <willy@...radead.org>,
        Mikulas Patocka <mpatocka@...hat.com>,
        Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [RFC PATCH] kbuild: Add option to generate a Compilation Database

On Fri, Jun 7, 2019 at 5:30 AM Raul E Rangel <rrangel@...omium.org> wrote:
>
> Clang tooling requires a compilation database to figure out the build
> options for each file. This enables tools like clang-tidy and
> clang-check.
>
> See https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html for more
> information.
>
> Normally cmake is used to generate the compilation database, but the
> linux kernel uses make. Another option is using
> [BEAR](https://github.com/rizsotto/Bear) which instruments
> exec to find clang invocations and generate the database that way.
>
> Clang 4.0.0 added the -MJ option to generate the json for each
> compilation unit. https://reviews.llvm.org/D27140
>
> This patch takes advantage of the -MJ option. So it only works for
> Clang.
>
> Signed-off-by: Raul E Rangel <rrangel@...omium.org>
> ---
> I have a couple TODOs in the code that I would like some feedback on.
> Specifically why extra-y doesn't seem to work in the root Makefile.
> Also, is there a way to add the correct list of prerequisites to the
> compile_commands.json target?
>
> Thanks,
> Raul
>
>
>  Makefile               | 20 ++++++++++++++++++++
>  lib/Kconfig.debug      |  7 +++++++
>  scripts/Makefile.build |  9 ++++++++-
>  3 files changed, 35 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index a61a95b6b38f7..06067ee18ff64 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1663,6 +1663,26 @@ quiet_cmd_tags = GEN     $@
>  tags TAGS cscope gtags: FORCE
>         $(call cmd,tags)
>
> +# Compilation Database
> +# ---------------------------------------------------------------------------
> +# Generates a compilation database that can be used with the LLVM tools
> +ifdef CONFIG_COMPILATION_DATABASE
> +
> +quiet_cmd_compilation_db = GEN   $@
> +cmd_compilation_db = (echo '['; \
> +       find "$(@D)" -mindepth 2 -iname '*.json' -print0 | xargs -0 cat; \
> +       echo ']') > "$(@D)/$(@F)"


Using 'find' has the same problem as
scripts/gen_compile_commands.py does.

Unless we start build from the clean source tree,
compile_commands.json will contain stale entries
from the previous building.

The motivation is
to replace scripts/gen_compile_commands.py entirely?

Generally, we do not need two ways to do the same thing...





> +# Make sure the database is built when calling `make` without a target.
> +# TODO: Using extra-y doesn't seem to work.
> +_all: $(obj)/compile_commands.json
> +
> +# TODO: Is there a variable that contains all the object files created by
> +# cmd_cc_o_c? Depending on `all` is kind of a hack
> +$(obj)/compile_commands.json: all FORCE
> +       $(call if_changed,compilation_db)
> +endif
> +
>  # Scripts to check various things for consistency
>  # ---------------------------------------------------------------------------
>
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index eae43952902eb..46fceb1fff3d9 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -238,6 +238,13 @@ config GDB_SCRIPTS
>           instance. See Documentation/dev-tools/gdb-kernel-debugging.rst
>           for further details.
>
> +config COMPILATION_DATABASE
> +       bool "Generate a compilation database"
> +       depends on CLANG_VERSION >= 40000
> +       help
> +         This creates a JSON Compilation Database (compile_commands.json)
> +         that is used by the clang tooling (clang-tidy, clang-check, etc).
> +
>  config ENABLE_MUST_CHECK
>         bool "Enable __must_check logic"
>         default y
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index ae9cf740633e1..0017bf397292d 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -145,8 +145,15 @@ $(obj)/%.ll: $(src)/%.c FORCE
>  # The C file is compiled and updated dependency information is generated.
>  # (See cmd_cc_o_c + relevant part of rule_cc_o_c)
>
> +ifdef CONFIG_COMPILATION_DATABASE
> +# TODO: Should we store the json in a temp variable and only copy it to the
> +# final name when the content is different? In theory we could avoid having to
> +# generate the compilation db if the json did not change.
> +compdb_flags = -MJ $(@D)/.$(@F).json
> +endif
> +
>  quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
> -      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
> +      cmd_cc_o_c = $(CC) $(c_flags) $(compdb_flags) -c -o $@ $<
>
>  ifdef CONFIG_MODVERSIONS
>  # When module versioning is enabled the following steps are executed:
> --
> 2.22.0.rc1.311.g5d7573a151-goog
>


-- 
Best Regards
Masahiro Yamada