| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20250916123156.4434aa19@foz.lan>
Date: Tue, 16 Sep 2025 12:31:56 +0200
From: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
To: Linux Doc Mailing List <linux-doc@...r.kernel.org>, Jonathan Corbet
<corbet@....net>
Cc: linux-kernel@...r.kernel.org
Subject: Re: [PATCH v6 00/21] Split sphinx call logic from docs Makefile
Em Tue, 16 Sep 2025 12:22:36 +0200
Mauro Carvalho Chehab <mchehab+huawei@...nel.org> escreveu:
> Hi Jon,
>
> This series should probably be called:
>
> "Move the trick-or-treat build hacks accumulated over time
> into a single place and document them."
>
> as this reflects its main goal. As such:
>
> - it places the jobserver logic on a library;
> - it removes sphinx/parallel-wrapper.sh;
> - the code now properly implements a jobserver-aware logic
> to do the parallelism when called via GNU make, failing back to
> "-j" when there's no jobserver;
> - converts check-variable-fonts.sh to Python and uses it via
> function call;
> - drops an extra script to generate man pages, adding a makefile
> target for it;
> - ensures that return code is 0 when PDF successfully builds;
> - about half of the script is comments and documentation.
>
> I tried to do my best to document all tricks that are inside the
> script. This way, the docs build steps is now documented.
>
> It should be noticed that it is out of the scope of this series
> to change the implementation. Surely the process can be improved,
> but first let's consolidate and document everything on a single
> place.
>
> Such script was written in a way that it can be called either
> directly or via a Makefile. Running outside Makefile is
> interesting specially when debug is needed. The command line
> interface replaces the need of having lots of env vars before
> calling sphinx-build:
>
> $ ./tools/docs/sphinx-build-wrapper --help
> usage: sphinx-build-wrapper [-h]
> [--sphinxdirs SPHINXDIRS [SPHINXDIRS ...]] [--conf CONF]
> [--builddir BUILDDIR] [--theme THEME] [--css CSS] [--paper {,a4,letter}] [-v]
> [-j JOBS] [-i] [-V [VENV]]
> {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}
>
> Kernel documentation builder
>
> positional arguments:
> {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}
> Documentation target to build
>
> options:
> -h, --help show this help message and exit
> --sphinxdirs SPHINXDIRS [SPHINXDIRS ...]
> Specific directories to build
> --conf CONF Sphinx configuration file
> --builddir BUILDDIR Sphinx configuration file
> --theme THEME Sphinx theme to use
> --css CSS Custom CSS file for HTML/EPUB
> --paper {,a4,letter} Paper size for LaTeX/PDF output
> -v, --verbose place build in verbose mode
> -j, --jobs JOBS Sets number of jobs to use with sphinx-build
> -i, --interactive Change latex default to run in interactive mode
> -V, --venv [VENV] If used, run Sphinx from a venv dir (default dir: sphinx_latest)
>
> the only mandatory argument is the target, which is identical with
> "make" targets.
>
> The call inside Makefile doesn't use the last four arguments. They're
> there to help identifying problems at the build:
>
> -v makes the output verbose;
> -j helps to test parallelism;
> -i runs latexmk in interactive mode, allowing to debug PDF
> build issues;
> -V is useful when testing it with different venvs.
>
> When used with GNU make (or some other make which implements jobserver),
> a call like:
>
> make -j <targets> htmldocs
>
> will make the wrapper to automatically use POSIX jobserver to claim
> the number of available job slots, calling sphinx-build with a
> "-j" parameter reflecting it. ON such case, the default can be
> overriden via SPHINXDIRS argument.
>
> Visiable changes when compared with the old behavior:
Visiable -> Visible
> When V=0, the only visible difference is that:
> - pdfdocs target now returns 0 on success, 1 on failures.
> This addresses an issue over the current process where we
> it always return success even on failures;
> - it will now print the name of PDF files that failed to build,
> if any.
>
> In verbose mode, sphinx-build-wrapper and sphinx-build command lines
> are now displayed.
It will also print a summary of all PDF built files, for pdfdocs
target in verbose mode.
>
> ---
>
> v6:
> - On success, PDF output is identical as before when V=0;
> - when V=1 is used, PDF output will print a build summary,
> as on v5;
> - solved a problem when multiple PDF files have the same
> basename but are located on different directories;
> - merged a patch series converting check-variable-fonts.sh
> to Python. Its logic is now called directly without running
> a subprocess.
> - venv patch moved to the end.
>
> v5:
> - merged comments with the script;
> - placed n_jobs on a separate function;
> - nitpick: dropped a for loop used instead of list append.
>
> v4:
> - updated references for sphinx-pre-install after its rename;
> - added some extra patches to add more options to python_version,
> allowing it to bail out and suggest alternatives;
> - added a patch at the end to explicitly break doc builds when
> python3 points to python3.6 or older.
>
> v3:
> - rebased on the top of docs-next;
> - added two patches to build man files that were on a separate
> patch series.
>
> v2:
> - there's no generic exception handler anymore;
> - it moves sphinx-pre-install to tools/docs;
> - the logic which ensures a minimal Python version got moved
> to a library, which is now used by both pre-install and wrapper;
> - The first wrapper (05/13) doesn't contain comments (except for
> shebang and SPDX). The goal is to help showing the size increase
> when moving from Makefile to Python. Some file increase is
> unavoidable, as Makefile is more compact: no includes, multple
> statements per line, no argparse, etc;
> - The second patch adds docstrings and comments. It has almost
> the same size of the code itself;
> - I moved the venv logic to a third wrapper patch;
> - I fixed an issue at the paraller build logic;
> - There are no generic except blocks anymore.
>
>
> Mauro Carvalho Chehab (21):
> scripts/jobserver-exec: move the code to a class
> scripts/jobserver-exec: move its class to the lib directory
> scripts/jobserver-exec: add a help message
> scripts: check-variable-fonts.sh: convert to Python
> tools/docs: check-variable-fonts.py: split into a lib and an exec file
> scripts: sphinx-pre-install: move it to tools/docs
> tools/docs: python_version: move version check from sphinx-pre-install
> tools/docs: python_version: drop a debug print
> tools/docs: python_version: allow check for alternatives and bail out
> tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
> docs: parallel-wrapper.sh: remove script
> docs: Makefile: document latex/PDF PAPER= parameter
> tools/docs: sphinx-build-wrapper: add an argument for LaTeX
> interactive mode
> tools/docs,scripts: sphinx-*: prevent sphinx-build crashes
> tools/docs: sphinx-build-wrapper: allow building PDF files in parallel
> tools/docs: sphinx-build-wrapper: Fix output for duplicated names
> docs: add support to build manpages from kerneldoc output
> tools: kernel-doc: add a see also section at man pages
> scripts: kdoc_parser.py: warn about Python version only once
> tools/docs: sphinx-* break documentation bulds on openSUSE
> tools/docs: sphinx-build-wrapper: add support to run inside venv
>
> Documentation/Makefile | 136 +--
> Documentation/doc-guide/kernel-doc.rst | 29 +-
> Documentation/doc-guide/sphinx.rst | 4 +-
> Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
> Documentation/sphinx/parallel-wrapper.sh | 33 -
> .../translations/it_IT/doc-guide/sphinx.rst | 4 +-
> .../translations/zh_CN/doc-guide/sphinx.rst | 4 +-
> Documentation/translations/zh_CN/how-to.rst | 2 +-
> MAINTAINERS | 4 +-
> Makefile | 2 +-
> scripts/check-variable-fonts.sh | 115 ---
> scripts/jobserver-exec | 88 +-
> scripts/lib/jobserver.py | 149 ++++
> scripts/lib/kdoc/kdoc_files.py | 5 +-
> scripts/lib/kdoc/kdoc_output.py | 84 +-
> scripts/lib/kdoc/kdoc_parser.py | 7 +-
> scripts/split-man.pl | 28 -
> tools/docs/check-variable-fonts.py | 23 +
> tools/docs/lib/latex_fonts.py | 162 ++++
> tools/docs/lib/python_version.py | 178 ++++
> tools/docs/sphinx-build-wrapper | 791 ++++++++++++++++++
> {scripts => tools/docs}/sphinx-pre-install | 135 +--
> 22 files changed, 1502 insertions(+), 483 deletions(-)
> delete mode 100644 Documentation/sphinx/parallel-wrapper.sh
> delete mode 100755 scripts/check-variable-fonts.sh
> create mode 100755 scripts/lib/jobserver.py
> delete mode 100755 scripts/split-man.pl
> create mode 100755 tools/docs/check-variable-fonts.py
> create mode 100755 tools/docs/lib/latex_fonts.py
> create mode 100644 tools/docs/lib/python_version.py
> create mode 100755 tools/docs/sphinx-build-wrapper
> rename {scripts => tools/docs}/sphinx-pre-install (93%)
>
Thanks,
Mauro
Powered by blists - more mailing lists