linux-kernel - Re: Parallel compilation performance regression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CA+icZUWTTdR42atb9FNYa90wwuvfFngksBWJBn2BLarkJsztxA@mail.gmail.com>
Date:   Sat, 20 Jun 2020 09:44:12 +0200
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     "Derrick, Jonathan" <jonathan.derrick@...el.com>
Cc:     "willy@...radead.org" <willy@...radead.org>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "dhowells@...hat.com" <dhowells@...hat.com>
Subject: Re: Parallel compilation performance regression

On Sat, Jun 20, 2020 at 5:53 AM Derrick, Jonathan
<jonathan.derrick@...el.com> wrote:
>
> On Thu, 2020-06-18 at 18:05 -0700, Matthew Wilcox wrote:
> > On Thu, Jun 18, 2020 at 11:52:55PM +0000, Derrick, Jonathan wrote:
> > > Hi David,
> > >
> > > I've been experiencing a performance regression when running a parallel
> > > compilation (eg, make -j72) on recent kernels.
> >
> > I bet you're using a version of make which predates 4.3:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0ddad21d3e99c743a3aa473121dc5561679e26bb
> >
>
> I am!
>
> # make --version
> GNU Make 4.2.1
>
>
> Thank you Matthew!

Check your distribution and included patches on top of a vanilla make v4.2.
Debian had some important ones I described in [1].

When proposing "make version 4.3" for tc-build - an opportunity to
build a llvm-toolchain the easy way - we saw different numbers.
I pointed in [1] to the Linus patch Matthew did here.

Personally, with switching to Debian's make version 4.3-3 I have seen
no big differences when using "make -j3" to build Linux v5.7+.
That might be different with "make -j72"...
...can I have SSH access to this machine, please :-),

You forgot to tell which Linux version you use.

If you are interested please look at closed tc-build issue #72 for our analysis.
For tc-build it did not matter - (Debian's) make v4.2 had some
slightly better performance.

BTW, with that pipe improvements in Linux v5.7 I see some better
numbers when using pipebench to benchmark my devices:

Example: SanDisk iSSD 16GB

root# cat /dev/sdb | pipebench > /dev/null
Summary:
Piped   14.91 GB in 00h01m25.20s:  179.23 MB/second

Before: approx. 100MB/s

BTW, I heard of hyperfine benchmark tool the first time when dealing
with make performance (see [2]) in ClangBuiltLinux.
I would like to see some benchmark numbers with hyperfine from you if
you do not mind.

Stay OPEN minded and curious.

Thanks.

- Sedat -

[1] https://github.com/ClangBuiltLinux/tc-build/issues/72
[2] https://github.com/sharkdp/hyperfine
[3] https://github.com/sharkdp/hyperfine/releases