linux-kernel - Re: [PATCH v2] docs: Use make invocation's -j argument for parallelism

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190924134325.4788fc30@coco.lan>
Date:   Tue, 24 Sep 2019 13:43:25 -0300
From:   Mauro Carvalho Chehab <mchehab+samsung@...nel.org>
To:     Kees Cook <keescook@...omium.org>
Cc:     Jonathan Corbet <corbet@....net>, linux-doc@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2] docs: Use make invocation's -j argument for
 parallelism

Em Mon, 23 Sep 2019 15:40:41 -0700
Kees Cook <keescook@...omium.org> escreveu:

> On Sun, Sep 22, 2019 at 02:03:31PM -0600, Jonathan Corbet wrote:
> > On Thu, 19 Sep 2019 14:44:37 -0700
> > Kees Cook <keescook@...omium.org> wrote:
> >   
> > > While sphinx 1.7 and later supports "-jauto" for parallelism, this
> > > effectively ignores the "-j" flag used in the "make" invocation, which
> > > may cause confusion for build systems. Instead, extract the available  
> > 
> > What sort of confusion might we expect?  Or, to channel akpm, "what are the
> > user-visible effects of this bug"?  
> 
> When I run "make htmldocs -j16" with a pre-1.7 sphinx, it is not
> parallelized.

Sphinx supports parallel builds for a while. With pre-1.7, you could do
something like:

	make SPHINXOPTS="-j16" htmldocs

Yet, on my experiences on big machines (tested here with Xeon with 40 and 64
CPU threads), parallel build doesn't actually benefit with values higher than
-j5 to -j8, with pre-1.7.

Sphinx 1.7 and higher seem to have improved a lot with "-jauto"
(although I didn't time it comparing with -j5 or -j8 on a big server).

> When I run "make htmldocs -j8" with 1.7+ sphinx, it uses
> all my CPUs instead of 8. :)

This should do the trick:

	make SPHINXOPTS="-j8" htmldocs

But yeah, IMHO, the best is if it could honor the Makefile flag:

	make -j8 htmldocs

If SPHINXOPTS doesn't contain "-j".

> > > +	-j $(shell python3 $(srctree)/scripts/jobserver-count $(SPHINX_PARALLEL)) \  
> > 
> > This (and the shebang line in the script itself) will cause the docs build
> > to fail on systems lacking Python 3.  While we have talked about requiring
> > Python 3 for the docs build, we have not actually taken that step yet.  We
> > probably shouldn't sneak it in here.  I don't see anything in the script
> > that should require a specific Python version, so I think it should be
> > tweaked to be version-independent and just invoke "python".  
> 
> Ah, no problem. I can fix this. In a quick scan it looked like sphinx
> was python3, but I see now that's just my install. :)

On Fedora 30, both python2 and python3 versions are available:

	python2-sphinx.noarch : Python documentation generator
	python3-sphinx.noarch : Python documentation generator

However, if one tries to install it without specifying "2" or "3", it
defaults to python2 version:

	$ sudo dnf install python-sphinx
	...
	Installing:
	 python2-sphinx                  noarch 1:1.8.4-1.fc30            fedora  1.8 M

AFAIKT, this also applies when distro upgrades takes place: upgrading
a python2 sphinx from Fedora 30 to Rawhide will very likely keep
the python2 version.

Thanks,
Mauro