linux-kernel - Re: [PATCH] gen_compile_commands: prune some directories

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 11 Feb 2021 10:19:08 -0700
From:   Nathan Chancellor <nathan@...nel.org>
To:     Masahiro Yamada <masahiroy@...nel.org>
Cc:     linux-kbuild@...r.kernel.org,
        Nathan Chancellor <natechancellor@...il.com>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        clang-built-linux@...glegroups.com, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] gen_compile_commands: prune some directories

On Fri, Feb 12, 2021 at 01:11:54AM +0900, Masahiro Yamada wrote:
> If directories are passed to gen_compile_commands.py, os.walk() traverses
> all the subdirectories to search for .cmd files, but we know some of them
> are not worth traversing.
> 
> Use the 'topdown' parameter of os.walk to prune them.
> 
> Documentation about the 'topdown' option of os.walk:
>   When topdown is True, the caller can modify the dirnames list
>   in-place (perhaps using del or slice assignment), and walk() will
>   only recurse into the subdirectories whose names remain in dirnames;
>   this can be used to prune the search, impose a specific order of
>   visiting, or even to inform walk() about directories the caller
>   creates or renames before it resumes walk() again. Modifying
>   dirnames when topdown is False has no effect on the behavior of
>   the walk, because in bottom-up mode the directories in dirnames
>   are generated before dirpath itself is generated.
> 
> This commit prunes four directories, .git, Documentation, include, and
> tools.
> 
> The first three do not contain any C files. My main motivation is the
> last one, tools/ directory.
> 
> Commit 6ca4c6d25949 ("gen_compile_commands: do not support .cmd files
> under tools/ directory") stopped supporting the tools/ directory.
> The current code no longer picks up .cmd files from the tools/
> directory.
> 
> If you run:
> 
>   ./scripts/clang-tools/gen_compile_commands.py --log_level=INFO
> 
> then, you will see several "File ... not found" log messages.
> 
> This is expected, and I do not want to support the tools/ directory.
> However, without an explicit comment "do not support tools/", somebody
> might try to get it back. Clarify this.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@...nel.org>

Sorry, I did not realize that gen_compile_commands.py did not intend to
support. I was only looking at the history for the current location, not
the former one of scripts/gen_compile_commands.py.

Acked-by: Nathan Chancellor <nathan@...nel.org>

> ---
> 
>  scripts/clang-tools/gen_compile_commands.py | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
> index 19963708bcf8..eb5faefbdf74 100755
> --- a/scripts/clang-tools/gen_compile_commands.py
> +++ b/scripts/clang-tools/gen_compile_commands.py
> @@ -20,7 +20,9 @@ _DEFAULT_LOG_LEVEL = 'WARNING'
>  _FILENAME_PATTERN = r'^\..*\.cmd$'
>  _LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$'
>  _VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
> -
> +# The tools/ directory adopts a different build system, and produces .cmd
> +# files in a different format. Do not support it.
> +_EXCLUDE_DIRS = ['.git', 'Documentation', 'include', 'tools']
>  
>  def parse_arguments():
>      """Sets up and parses command-line arguments.
> @@ -80,8 +82,14 @@ def cmdfiles_in_dir(directory):
>      """
>  
>      filename_matcher = re.compile(_FILENAME_PATTERN)
> +    exclude_dirs = [ os.path.join(directory, d) for d in _EXCLUDE_DIRS ]
> +
> +    for dirpath, dirnames, filenames in os.walk(directory, topdown=True):
> +        # Prune unwanted directories.
> +        if dirpath in exclude_dirs:
> +            dirnames[:] = []
> +            continue
>  
> -    for dirpath, _, filenames in os.walk(directory):
>          for filename in filenames:
>              if filename_matcher.match(filename):
>                  yield os.path.join(dirpath, filename)
> -- 
> 2.27.0
>