linux-kernel - Re: [PATCH] docs: Use make invocation's -j argument for parallelism

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190919162549.5ccdab62@coco.lan>
Date:   Thu, 19 Sep 2019 16:25:49 -0300
From:   Mauro Carvalho Chehab <mchehab+samsung@...nel.org>
To:     Kees Cook <keescook@...omium.org>
Cc:     Jonathan Corbet <corbet@....net>, linux-doc@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH] docs: Use make invocation's -j argument for parallelism

Em Thu, 19 Sep 2019 11:18:52 -0700
Kees Cook <keescook@...omium.org> escreveu:

> While sphinx 1.7 and later supports "-jauto" for parallelism, this
> effectively ignores the "-j" flag used in the "make" invocation, which
> may cause confusion for build systems. Instead, extract the available
> parallelism from "make"'s job server (since it is not exposed in any
> special variables) and use that for the "sphinx-build" run. Now things
> work correctly for builds where -j is specified at the top-level:
> 
> 	make -j16 htmldocs
> 
> Signed-off-by: Kees Cook <keescook@...omium.org>
> ---
>  Documentation/Makefile  |  3 +--
>  scripts/jobserver-count | 46 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 47 insertions(+), 2 deletions(-)
>  create mode 100755 scripts/jobserver-count
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index e145e4db508b..4408eeaf2891 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -33,8 +33,6 @@ ifeq ($(HAVE_SPHINX),0)
>  
>  else # HAVE_SPHINX
>  
> -export SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while (<IN>) { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN')
> -
>  # User-friendly check for pdflatex and latexmk
>  HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
>  HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
> @@ -68,6 +66,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
>  	PYTHONDONTWRITEBYTECODE=1 \
>  	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
>  	$(SPHINXBUILD) \
> +	-j $(shell $(srctree)/scripts/jobserver-count) \
>  	-b $2 \
>  	-c $(abspath $(srctree)/$(src)) \
>  	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
> diff --git a/scripts/jobserver-count b/scripts/jobserver-count
> new file mode 100755
> index 000000000000..5fc9d2fd5254
> --- /dev/null
> +++ b/scripts/jobserver-count
> @@ -0,0 +1,46 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# This determines how many parallel tasks "make" is expecting, as it is
> +# not exposed via an special variables.
> +# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> +import os, sys, fcntl
> +
> +# Set non-blocking for a given file descriptor.
> +def nonblock(fd):
> +	flags = fcntl.fcntl(fd, fcntl.F_GETFL)
> +	fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
> +	return fd
> +


> +# Fetch the make environment options.
> +flags = os.environ.get('MAKEFLAGS', None)
> +if flags == None:
> +	print("1")
> +	sys.exit(0)
> +
> +# Look for "--jobserver=R,W"
> +opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
> +if len(opts) != 1:
> +	print("1")
> +	sys.exit(0)

Using "1" as default if -j is not specified doesn't sound a good idea. I 
mean, Sphinx is very slow without any parallelism. I would keep "auto" as
default here, if version >= 1.7 (and if there's no explicit SPHINXOPTS).


> +
> +# Parse out R,W file descriptor numbers and set them nonblocking.
> +fds = opts[0].split("=", 1)[1]
> +reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)]
> +
> +# Read out as many jobserver slots as possible.
> +jobs = b""
> +while True:
> +	try:
> +		slot = os.read(reader, 1)
> +		jobs += slot
> +	except:
> +		break
> +# Return all the reserved slots.
> +os.write(writer, jobs)
> +
> +# Report available slots (with a bump for our caller's reserveration).
> +if len(jobs) < 1:
> +	print("1")
> +else:
> +	print(len(jobs) + 1)



Thanks,
Mauro