linux-kernel - Re: [PATCH v2] builddeb: parallelize debug module installation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150428122258.GA2848@canonical.com>
Date:	Tue, 28 Apr 2015 07:22:59 -0500
From:	Chris J Arges <chris.j.arges@...onical.com>
To:	Michal Marek <mmarek@...e.cz>
Cc:	Riku Voipio <riku.voipio@...aro.org>,
	linux-kbuild <linux-kbuild@...r.kernel.org>,
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2] builddeb: parallelize debug module installation

On Tue, Apr 28, 2015 at 12:05:19PM +0200, Michal Marek wrote:
> On 2015-04-28 10:57, Riku Voipio wrote:
> > On 27 April 2015 at 19:43, Chris J Arges <chris.j.arges@...onical.com> wrote:
> >> When building the dbg package, we use a large 'for module in $(find' loop that
> >> can be easily parallelized by using 'find | xargs'. This patch modifies this
> >> loop to use the later paradigm.
> >>
> >> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
> >> possible.
> >>
> >> Signed-off-by: Chris J Arges <chris.j.arges@...onical.com>
> >> ---
> >>  scripts/package/builddeb | 15 ++++++++-------
> >>  1 file changed, 8 insertions(+), 7 deletions(-)
> >>
> >> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> >> index 88dbf23..538f829 100755
> >> --- a/scripts/package/builddeb
> >> +++ b/scripts/package/builddeb
> >> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
> >>                 rmdir "$tmpdir/lib/modules/$version"
> >>         fi
> >>         if [ -n "$BUILD_DEBUG" ] ; then
> >> -               for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> >> -                       module=lib/modules/$module
> >> -                       mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> >> +               find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
> > 
> > I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
> > of modules (allmodconfig will make 4500).
> 
> Yep. I was thinking about retrieving the value of the make -j argument
> somehow, but this is not possible and we certainly do not want to
> implement the jobserver protocol in shell :-). So using the number of
> processors is a sensible choice. What can be done is to detect whether
> the -j option is in MAKEFLAGS and only then use multiple instances.
> 
> Michal
>

Michal,

The MAKEFLAGS variable will have the jobserver fd's in it, if we are building
in parallel with -j. I could use this to detect if we are building in parallel
and adjust the '-P' flag in xargs accordingly. I'll work on v3 of this patch,
thanks for the reviews!

--chris 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/