linux-kernel - Re: [PATCH] kbuild: use smaller dictionary size for zstd module compression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAK7LNATBvZBmuznFaFQgZeOO_F_UsisPYBobDC7LYDMVW3Yd=w@mail.gmail.com>
Date:   Sun, 19 Sep 2021 20:36:11 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     Piotr Górski <lucjan.lucjanov@...il.com>
Cc:     Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Michal Marek <michal.lkml@...kovi.net>,
        Nick Desaulniers <ndesaulniers@...gle.com>
Subject: Re: [PATCH] kbuild: use smaller dictionary size for zstd module compression

On Sat, Sep 18, 2021 at 12:07 AM Piotr Górski <lucjan.lucjanov@...il.com> wrote:
>
> Ping...
>
> pt., 10 wrz 2021 o 15:10 Piotr Gorski <lucjan.lucjanov@...il.com> napisał(a):
> >
> > By default, zstd without parameters uses a dictionary size of 8 MB.
> > However, most modules are much smaller than that.
> > Use a dictionary size of 2 MB for module compression, resulting in
> > slightly higher compression speed while still maintaining a good
> > compression ratio.
> > The --zstd=wlog=21 option is equivalent to --lzma2=dict=2MiB used in XZ compression.



What is a benefit of this patch?

It claims "slightly higher compression", but I do not see a difference.
(15.0577 sec   --> 15.013 sec)


[1] build  allmodconfig
[2] collect *.ko files whose size is less than 2MB
[3] measure the time elapsed for compressing all of them

I did 'perf stat --repeat 10'.






The default zstd:



 Performance counter stats for '/tmp/zstd2/test.sh' (10 runs):

         19,250.23 msec task-clock                #    1.278 CPUs
utilized            ( +-  0.49% )
            65,424      context-switches          #    0.003 M/sec
               ( +-  0.03% )
             5,609      cpu-migrations            #    0.291 K/sec
               ( +-  0.25% )
         4,207,419      page-faults               #    0.219 M/sec
               ( +-  0.19% )
    54,349,495,038      cycles                    #    2.823 GHz
               ( +-  0.15% )  (85.93%)
     3,833,225,032      stalled-cycles-frontend   #    7.05% frontend
cycles idle     ( +-  0.47% )  (86.34%)
    20,789,203,221      stalled-cycles-backend    #   38.25% backend
cycles idle      ( +-  0.31% )  (85.92%)
    65,385,071,612      instructions              #    1.20  insn per
cycle
                                                  #    0.32  stalled
cycles per insn  ( +-  0.09% )  (86.54%)
    11,033,842,879      branches                  #  573.180 M/sec
               ( +-  0.10% )  (86.38%)
       288,314,097      branch-misses             #    2.61% of all
branches          ( +-  0.16% )  (86.33%)

           15.0577 +- 0.0728 seconds time elapsed  ( +-  0.48% )



zstd --zstd=wlog=21:




 Performance counter stats for '/tmp/zstd2/test2.sh' (10 runs):

         19,203.08 msec task-clock                #    1.279 CPUs
utilized            ( +-  0.85% )
            65,404      context-switches          #    0.003 M/sec
               ( +-  0.05% )
             5,617      cpu-migrations            #    0.293 K/sec
               ( +-  0.15% )
         4,214,502      page-faults               #    0.219 M/sec
               ( +-  0.12% )
    54,495,218,169      cycles                    #    2.838 GHz
               ( +-  0.29% )  (85.55%)
     3,867,166,903      stalled-cycles-frontend   #    7.10% frontend
cycles idle     ( +-  0.26% )  (87.22%)
    20,950,577,113      stalled-cycles-backend    #   38.44% backend
cycles idle      ( +-  0.38% )  (86.51%)
    65,500,859,667      instructions              #    1.20  insn per
cycle
                                                  #    0.32  stalled
cycles per insn  ( +-  0.12% )  (86.65%)
    11,045,504,296      branches                  #  575.194 M/sec
               ( +-  0.12% )  (85.90%)
       288,729,169      branch-misses             #    2.61% of all
branches          ( +-  0.12% )  (85.59%)

            15.013 +- 0.129 seconds time elapsed  ( +-  0.86% )






> > Signed-off-by: Piotr Gorski <lucjan.lucjanov@...il.com>
> > ---
> >  scripts/Makefile.modinst | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> > index ff9b09e4cfca..c3475e5aca7c 100644
> > --- a/scripts/Makefile.modinst
> > +++ b/scripts/Makefile.modinst
> > @@ -97,7 +97,7 @@ quiet_cmd_gzip = GZIP    $@
> >  quiet_cmd_xz = XZ      $@
> >        cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
> >  quiet_cmd_zstd = ZSTD    $@
> > -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> > +      cmd_zstd = $(ZSTD) --zstd=wlog=21 -T0 --rm -f -q $<
> >
> >  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
> >         $(call cmd,gzip)
> > --
> > 2.33.0.142.ge0a2f5cbc5
> >



--
Best Regards
Masahiro Yamada