linux-kernel - [PATCH] gen_compile_commands: prune some directories

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20210211161154.3892836-1-masahiroy@kernel.org>
Date:   Fri, 12 Feb 2021 01:11:54 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     linux-kbuild@...r.kernel.org
Cc:     Masahiro Yamada <masahiroy@...nel.org>,
        Nathan Chancellor <natechancellor@...il.com>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        clang-built-linux@...glegroups.com, linux-kernel@...r.kernel.org
Subject: [PATCH] gen_compile_commands: prune some directories

If directories are passed to gen_compile_commands.py, os.walk() traverses
all the subdirectories to search for .cmd files, but we know some of them
are not worth traversing.

Use the 'topdown' parameter of os.walk to prune them.

Documentation about the 'topdown' option of os.walk:
  When topdown is True, the caller can modify the dirnames list
  in-place (perhaps using del or slice assignment), and walk() will
  only recurse into the subdirectories whose names remain in dirnames;
  this can be used to prune the search, impose a specific order of
  visiting, or even to inform walk() about directories the caller
  creates or renames before it resumes walk() again. Modifying
  dirnames when topdown is False has no effect on the behavior of
  the walk, because in bottom-up mode the directories in dirnames
  are generated before dirpath itself is generated.

This commit prunes four directories, .git, Documentation, include, and
tools.

The first three do not contain any C files. My main motivation is the
last one, tools/ directory.

Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
under tools/ directory") stopped supporting the tools/ directory.
The current code no longer picks up .cmd files from the tools/
directory.

If you run:

  ./scripts/clang-tools/gen_compile_commands.py --log_level=INFO

then, you will see several "File ... not found" log messages.

This is expected, and I do not want to support the tools/ directory.
However, without an explicit comment "do not support tools/", somebody
might try to get it back. Clarify this.

Signed-off-by: Masahiro Yamada <masahiroy@...nel.org>
---

 scripts/clang-tools/gen_compile_commands.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index 19963708bcf8..eb5faefbdf74 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -20,7 +20,9 @@ _DEFAULT_LOG_LEVEL = 'WARNING'
 _FILENAME_PATTERN = r'^\..*\.cmd$'
 _LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$'
 _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
-
+# The tools/ directory adopts a different build system, and produces .cmd
+# files in a different format. Do not support it.
+_EXCLUDE_DIRS = ['.git', 'Documentation', 'include', 'tools']

 def parse_arguments():
     """Sets up and parses command-line arguments.
@@ -80,8 +82,14 @@ def cmdfiles_in_dir(directory):
     """

     filename_matcher = re.compile(_FILENAME_PATTERN)
+    exclude_dirs = [ os.path.join(directory, d) for d in _EXCLUDE_DIRS ]
+
+    for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
+        # Prune unwanted directories.
+        if dirpath in exclude_dirs:
+            dirnames[:] = []
+            continue

-    for dirpath, _, filenames in os.walk(directory):
         for filename in filenames:
             if filename_matcher.match(filename):
                 yield os.path.join(dirpath, filename)
-- 
2.27.0