linux-kernel - Re: [PATCH 3/5] kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20230811112530.GA453409@lxhi-087>
Date:   Fri, 11 Aug 2023 13:25:30 +0200
From:   Eugeniu Rosca <erosca@...adit-jv.com>
To:     Masahiro Yamada <masahiroy@...nel.org>
CC:     Eugeniu Rosca <erosca@...adit-jv.com>,
        Nicolas Schier <n.schier@....de>,
        SzuWei Lin <szuweilin@...gle.com>,
        <linux-kbuild@...r.kernel.org>, <linux-kernel@...r.kernel.org>,
        <linux-arch@...r.kernel.org>, <Matthias.Thomae@...bosch.com>,
        <yyankovskyi@...adit-jv.com>, <Dirk.Behme@...bosch.com>,
        Eugeniu Rosca <roscaeugeniu@...il.com>
Subject: Re: [PATCH 3/5] kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}

Hello Yamada-san,

Appreciate your generous support, which allowed finding the root-cause.

On Wed, Aug 02, 2023 at 06:21:14PM +0900, Masahiro Yamada wrote:

[..]

> Indeed, reverting 7ce7e984ab2b218d6e92d5165629022fe2daf9ee
> makes qcom's external module build successfully
> (but rebuilding is super slow).

Same observation.

> 
> Interestingly, revert 7ce7e984ab2b218d6e92d5165629022fe2daf9ee
> then apply the attached patch, then
> 'Argument list too long' will come back.
> 
> So, this is unrelated to the actual build commands.
> 
> I suspect bare 'export', which expands all variables
> while apparently most of them are not meant exported.

Indeed, that seems to be the case and more evidence supports this:

 * Several pre-existing commits point out to the same root-cause:

    https://git.codelinaro.org/clo/la/platform/vendor/qcom/opensource/audio-kernel-ar/-/commit/4025a25a2479fc34
    ("makefile: kona: remove make export <all variables> instances")

    https://github.com/sonyxperiadev/kernel-techpack-audio/commit/02f00754120df2
    ("audio-kernel: Fix build time issue")

 * The more recent GNU Make 4.4.1 (called with -ddd) reveals more
    details just before hitting 'Argument list too long':

    scripts/Makefile.lib:431: not recursively expanding size_append to export to shell function
    scripts/Makefile.lib:447: not recursively expanding cmd_file_size to export to shell function
    scripts/Makefile.lib:453: not recursively expanding cmd_bzip2_with_size to export to shell function
    scripts/Makefile.lib:462: not recursively expanding cmd_lzma_with_size to export to shell function
    scripts/Makefile.lib:468: not recursively expanding cmd_lzo_with_size to export to shell function
    scripts/Makefile.lib:474: not recursively expanding cmd_lz4_with_size to export to shell function
    scripts/Makefile.lib:520: not recursively expanding cmd_xzkern_with_size to export to shell function
    scripts/Makefile.lib:549: not recursively expanding cmd_zstd22_with_size to export to shell function

    The "not recursively expanding" messages above come from GNU
    make commit https://gnu.googlesource.com/make/+/7d484017077089a
    ("[SV 63016] Don't fail exporting to $(shell ...)"), which appears
    to be saying that our issue is caused by exporting a makefile
    variable which contains the 'shell' directive in its definition.

    I think it would be too much of a burden for Kbuild not to make use
    of any variables using the 'shell' keyword, so I tend to agree that
    defending against bare 'export' statements in Kbuild is rather not
    feasible and it should be fixed in the makefiles of out-of-tree *ko.

> Insert the following in your reproducer, then it will work.

Thanks for the demo code snippet.

-- 
Best regards,
Eugeniu Rosca