linux-kernel - Re: Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CA+icZUWC-1uhVK52v=twDWSKqbBMAh=Fp0iXA0Acoq_AYxOnWQ@mail.gmail.com>
Date:   Fri, 9 Apr 2021 13:51:16 +0200
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Piotr Gorski <lucjan.lucjanov@...il.com>
Cc:     linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org,
        masahiroy@...nel.org, oleksandr@...alenko.name
Subject: Re: Subject: Re: [PATCH v3] kbuild: add support for zstd compressed modules

On Fri, Apr 9, 2021 at 1:31 PM Sedat Dilek <sedat.dilek@...il.com> wrote:
>
> On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski <lucjan.lucjanov@...il.com> wrote:
> >
> > I originally posted the patch in a different form [1] even before Masahiro's changes.
> > I've been testing this solution since December last year and posted it in March this year,
> > after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2]
> >
> > Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did.
> >
> > My current logs:
> >
> > lucjan@...hlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz
> > CONFIG_DEBUG_INFO=y
> > # CONFIG_DEBUG_INFO_REDUCED is not set
> > # CONFIG_DEBUG_INFO_COMPRESSED is not set
> > # CONFIG_DEBUG_INFO_SPLIT is not set
> > CONFIG_DEBUG_INFO_DWARF4=y
> > CONFIG_DEBUG_INFO_BTF=y
> > CONFIG_DEBUG_INFO_BTF_MODULES=y
> > lucjan@...hlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz
> > CONFIG_MODULE_COMPRESS_ZSTD=y
> > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19
> >
> > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use.
> >
> > The only difference - I don't use clang. Maybe those who use will comment on this.
> > I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression.
> >
> > [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2
> >
> > [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2
> >
> > [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2
>
> I am a big fan of ZSTD and have it as default in all available Linux
> Kconfigs and Debian's initramfs-tools.
> So, I am highly interested in getting this fixed.
>
> Unfortunately, I have thrown away my yesterday's Clang-LTO build and
> switched to Clang-CFI with builddeb - should do handle the same way.
>
> I see three iwlwifi.ko (as an example):
>
> $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 8.2M Apr  9 11:07
> drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB
> relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
> debug_info, not stripped
>
> That iwlwifi.ko with debug-info is optimized when moving to
> debian/linux-image-dbg directory:
>
> $ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 7.9M Apr  9 11:18
> debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
> ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
> debug_info, not stripped
>
> And think it's shrunk down and included debian/linux-image directory:
>
> $ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 694K Apr  9 11:18
> debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
> ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped
>
> I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed.
> Fact is the one with debug-info is done correctly.
> Might be builddeb script needs a special treatment.
>

OK, I see (sorry Gmail truncates my paste of snippet).

We need to add in this block a CONFIG_MODULE_COMPRESS_XXX handling:

[ scripts/package/builddeb ]

#159: if is_enabled CONFIG_MODULES; then
...
#184: fi

Maybe other scripts in scripts/package/ directory, too.

What do you say Masahiro?

I have to admit I never used any compression for kernel-modules before
and after recent changes in <kbuild.git#kbuild>.

- Sedat -

[1] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/tree/scripts/package/builddeb?h=kbuild#n159