linux-kernel - Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAK7LNAQ+2fhZNhuCi_40V=4QN+pMC7y67MTYp2zO7HOVwMJH_Q@mail.gmail.com>
Date:   Thu, 16 Mar 2023 01:39:48 +0900
From:   Masahiro Yamada <masahiroy@...nel.org>
To:     sedat.dilek@...il.com
Cc:     Bastian Germann <bage@...utronix.de>,
        Nathan Chancellor <nathan@...nel.org>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 1/1] kbuild: deb-pkg: Allow parallel build

On Thu, Mar 16, 2023 at 12:57 AM Sedat Dilek <sedat.dilek@...il.com> wrote:
>
> On Wed, Mar 15, 2023 at 3:24 PM Masahiro Yamada <masahiroy@...nel.org> wrote:
> >
> > On Tue, Mar 14, 2023 at 3:56 AM Bastian Germann <bage@...utronix.de> wrote:
> > >
> > > Am 13.03.23 um 18:59 schrieb Masahiro Yamada:
> > > > On Tue, Mar 14, 2023 at 2:10 AM Bastian Germann <bage@...utronix.de> wrote:
> > > >>
> > > >> Currently, the only way to build the deb-pkg generated package parallely
> > > >> is adding -jN to the MAKEFLAGS environment variable. The package ignores
> > > >> the usual parallel build option that is described in Debian Policy §4.9.1.
> > > >
> > > >
> > > >
> > > > "dpkg-buildpackage -b -j16" worked for me.
> > >
> > > This ends up in DEB_BUILD_OPTIONS=parallel=16 being set and the call:
> > > /usr/bin/make -f ./Makefile ARCH=x86    KERNELRELEASE=6.3.0-rc2 KBUILD_BUILD_VERSION=1  olddefconfig all
> >
> >
> > "dpkb-buildpackage -j<N>" sets not only DEB_BUILD_OPTIONS
> > but also MAKEFLAGS.
> >
> >
> > This is clearly explained in "man dpkb-buildpackage".
> >
> >
> >  -j, --jobs[=jobs|auto]
> >    ...
> >   Will add itself to the MAKEFLAGS environment variable, which should
> >   cause all subsequent make invocations to inherit the option, thus
> >   forcing the parallel setting on the packaging ...
> >
> >
> >
> >
> > Your statement sounds like
> >
> >  'MAKEFLAGS=-j<N> dpkg-buildpackage'
> >
> > is the only way to build packages in parallel.
> >
> > Apparently, dpkg-buildpackage provides a much shorter way
> > and invokes internal Make in parallel.
> >
> >
> >
> >
> > >
> > > So it is not used to run the actual build, just the top level `make -f debian/rules` invocation.
> > > You can set --jobs-force=16, which ends up in MAKEFLAGS but the point of the patch is that one can use
> > > the usual way of parallelizing.
> >
> >
> > What is the "usual" way in this context?
> >
> >
> > Do you mean
> > 'DEB_BUILD_OPTIONS=parallel=16 dpkg-buildpackage -b'
> > is the usual way for parallel building?
> >
> > If so, I agree. This patch caters to this case.
> >
> >
> > But, I think your code should go to debian/rules
> > instead of scripts/package/deb-build-option.
> > Kbuild's "make clean" works much faster with the parallel option.
> >
> >
> > Also, the commit description should explain the
> > benefit of this patch.
> >
> >
> > >
> > > Side note: Without the patch, the build is run with -j1 regardless of being called from the Linux Makefile
> > > or dpkg-buildpackage.
> >
> >
> > Try:
> >
> >  $ make deb-pkg -j16
> >
> > and
> >
> >  $ dpkg-buildpackage -b -j16
> >
> >
> > Both run the package building in parallel.
> >
> > You see a big difference in build speed w/wo the -j option.
> > (and the build logs are interleaved, since they
> > are emitted by multiple threads)
> >
> >
>
> I have no strong opinion in having a DEB_BUILD_OPTIONS...
>
> As you stated Masahiro I have seen it mostly in debian/rules.
>
> See the example from [1]:
>
> ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
>     NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
>     MAKEFLAGS += -j$(NUMJOBS)
> endif
>

Yup, this code looks good to me.




-- 
Best Regards
Masahiro Yamada