linux-kernel - [RFC PATCH 0/3] kbuild: generate intermediate C files instead of copying

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-Id: <1503132577-24423-1-git-send-email-yamada.masahiro@socionext.com>
Date:   Sat, 19 Aug 2017 17:49:34 +0900
From:   Masahiro Yamada <yamada.masahiro@...ionext.com>
To:     Sam Ravnborg <sam@...nborg.org>, Michal Marek <mmarek@...e.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        linux-kbuild@...r.kernel.org
Cc:     devicetree@...r.kernel.org, Rob Herring <robh@...nel.org>,
        Jonathan Corbet <corbet@....net>,
        Richard Purdie <richard.purdie@...uxfoundation.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        Mauro Carvalho Chehab <mchehab@...nel.org>,
        Nicholas Piggin <npiggin@...il.com>, linux-doc@...r.kernel.org,
        Markus Heiser <markus.heiser@...marit.de>,
        linux-kernel@...r.kernel.org,
        Frank Rowand <frowand.list@...il.com>,
        Rob Herring <robh+dt@...nel.org>,
        SeongJae Park <sj38.park@...il.com>,
        "Yann E. MORIN" <yann.morin.1998@...e.fr>
Subject: [RFC PATCH 0/3] kbuild: generate intermediate C files instead of copying _shipped files

In Linux build system convention, we do not run tools such as
flex, bison, gperf during the kernel building.  Instead, manage
generated C files in the repository with _shipped suffixes.
They are simply shipped (copied) removing the _shipped suffixes
during the kernel building.

Commit 7373f4f83c71 ("kbuild: add implicit rules for parser generation")
added a mechanism to regenerate intermediate C files easily.
The build rules are surrounded with ifdef REGENERATE_PARSERS.
So, we need to pass REGENERATE_PARSERS=1 from the command line
when we want to update them.

Here is one question.  Is it acceptable to use those rules all the time?
That is, generate those C files by flex, bison, gperf during the
kernel building.

This means, the build system depends on more external tools.
>From the users' point of view, they will need to install
flex, bison, gperf in order to build the kernel.
>From the developers' point of view, the advantage is
we do not need to version-control generated files, i.e. _shipped files
will be deleted.

I'd like to know if this is acceptable or not.

For example, currently some files are simply shipped (copied)
when building the kconfig program.

  $ make mrproper; make defconfig
    HOSTCC  scripts/basic/fixdep
    HOSTCC  scripts/kconfig/conf.o
    SHIPPED scripts/kconfig/zconf.tab.c
    SHIPPED scripts/kconfig/zconf.lex.c
    SHIPPED scripts/kconfig/zconf.hash.c
    HOSTCC  scripts/kconfig/zconf.tab.o
    HOSTLD  scripts/kconfig/conf
  *** Default configuration is based on 'x86_64_defconfig'
  #
  # configuration written to .config
  #

With this series, they are created from *real* sources
(*.y, *.l, *.gperf files).

  $ make mrproper; make defconfig
    HOSTCC  scripts/basic/fixdep
    HOSTCC  scripts/kconfig/conf.o
    YACC    scripts/kconfig/zconf.tab.c
    LEX     scripts/kconfig/zconf.lex.c
    GPERF   scripts/kconfig/zconf.hash.c
    HOSTCC  scripts/kconfig/zconf.tab.o
    HOSTLD  scripts/kconfig/conf
  *** Default configuration is based on 'x86_64_defconfig'
  #
  # configuration written to .config
  #

Note:
The tool versions in Documentation/process/changes.rst are just
place-holders for now.  We need to figure out the minimal versions
if we like to switch to this approach.



Masahiro Yamada (3):
  kbuild: generate *.hash.c during build
  kbuild: generate *.lex.c during build
  kbuild: generate *.tab.c and *.tab.h during build

 Documentation/process/changes.rst        |   36 +
 scripts/Makefile.lib                     |   26 +-
 scripts/dtc/Makefile                     |    6 +-
 scripts/dtc/dtc-lexer.lex.c_shipped      | 2259 ---------------------------
 scripts/dtc/dtc-parser.tab.c_shipped     | 2303 ----------------------------
 scripts/dtc/dtc-parser.tab.h_shipped     |  125 --
 scripts/genksyms/Makefile                |    4 +-
 scripts/genksyms/keywords.hash.c_shipped |  230 ---
 scripts/genksyms/lex.lex.c_shipped       | 2291 ---------------------------
 scripts/genksyms/parse.tab.c_shipped     | 2394 -----------------------------
 scripts/genksyms/parse.tab.h_shipped     |  119 --
 scripts/kconfig/Makefile                 |    1 +
 scripts/kconfig/zconf.hash.c_shipped     |  297 ----
 scripts/kconfig/zconf.lex.c_shipped      | 2473 ------------------------------
 scripts/kconfig/zconf.tab.c_shipped      | 2471 -----------------------------
 15 files changed, 53 insertions(+), 14982 deletions(-)
 delete mode 100644 scripts/dtc/dtc-lexer.lex.c_shipped
 delete mode 100644 scripts/dtc/dtc-parser.tab.c_shipped
 delete mode 100644 scripts/dtc/dtc-parser.tab.h_shipped
 delete mode 100644 scripts/genksyms/keywords.hash.c_shipped
 delete mode 100644 scripts/genksyms/lex.lex.c_shipped
 delete mode 100644 scripts/genksyms/parse.tab.c_shipped
 delete mode 100644 scripts/genksyms/parse.tab.h_shipped
 delete mode 100644 scripts/kconfig/zconf.hash.c_shipped
 delete mode 100644 scripts/kconfig/zconf.lex.c_shipped
 delete mode 100644 scripts/kconfig/zconf.tab.c_shipped

-- 
2.7.4