linux-kernel - Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+icZUV1ng9-TurSfKFuYueuHUbdkCxy3hsMOM9DrtTVL0SsHA@mail.gmail.com>
Date:   Wed, 15 Mar 2023 16:56:54 +0100
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Masahiro Yamada <masahiroy@...nel.org>
Cc:     Bastian Germann <bage@...utronix.de>,
        Nathan Chancellor <nathan@...nel.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@...nel.org> wrote:
>
> On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@...utronix.de> wrote:
> >
> > Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@...utronix.de> wrote:
> > >>
> > >> Currently, the only way to build the deb-pkg generated package parallely
> > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> > >> the usual parallel build option that is described in Debian Policy §4.9.1.
> > >
> > >
> > >
> > > "dpkg-buildpackage -b -j16" worked for me.
> >
> > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
> > /usr/bin/make -f ./Makefile ARCH=x86    KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1  olddefconfig all
>
>
> "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> but also MAKEFLAGS.
>
>
> This is clearly explained in "man dpkb-buildpackage".
>
>
>  -j, --jobs[=jobs|auto]
>    ...
>   Will add itself to the MAKEFLAGS environment variable, which should
>   cause all subsequent make invocations to inherit the option, thus
>   forcing the parallel setting on the packaging ...
>
>
>
>
> Your statement sounds like
>
>  'MAKEFLAGS=-j<N> dpkg-buildpackage'
>
> is the only way to build packages in parallel.
>
> Apparently, dpkg-buildpackage provides a much shorter way
> and invokes internal Make in parallel.
>
>
>
>
> >
> > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> > the usual way of parallelizing.
>
>
> What is the "usual" way in this context?
>
>
> Do you mean
> 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> is the usual way for parallel building?
>
> If so, I agree. This patch caters to this case.
>
>
> But, I think your code should go to debian/rules
> instead of scripts/package/deb-build-option.
> Kbuild's "make clean" works much faster with the parallel option.
>
>
> Also, the commit description should explain the
> benefit of this patch.
>
>
> >
> > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
> > or dpkg-buildpackage.
>
>
> Try:
>
>  $ make deb-pkg -j16
>
> and
>
>  $ dpkg-buildpackage -b -j16
>
>
> Both run the package building in parallel.
>
> You see a big difference in build speed w/wo the -j option.
> (and the build logs are interleaved, since they
> are emitted by multiple threads)
>
>

I have no strong opinion in having a DEB_BUILD_OPTIONS...

As you stated Masahiro I have seen it mostly in debian/rules.

See the example from [1]:

ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
    MAKEFLAGS += -j$(NUMJOBS)
endif

Normally, I use in my kernel build-script:

$ make deb-pkg -j${NUMJOBS}

My €0,02.

-Sedat-


[1] https://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options