linux-kernel - Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <9b3b0430-066f-486e-89cc-00e6f1f3b096@gmail.com>
Date: Sat, 16 Aug 2025 10:16:01 +0900
From: Akira Yokosawa <akiyks@...il.com>
To: mchehab+huawei@...nel.org
Cc: alex.gaynor@...il.com, aliceryhl@...gle.com, bjorn3_gh@...tonmail.com,
 boqun.feng@...il.com, corbet@....net, gary@...yguo.net,
 linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
 rust-for-linux@...r.kernel.org, tmgross@...ch.edu,
 Akira Yokosawa <akiyks@...il.com>
Subject: Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for
 sphinx-build

Hi Mauro,

On Fri, 15 Aug 2025 13:50:32 +0200, Mauro Carvalho Chehab wrote:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
> 
> Such script is designed to work both as an standalone command
> and as part of a Makefile. As such, it properly handles POSIX
> jobserver used by GNU make.
> 
> It should be noticed that, when running the script alone,
> it will only take care of sphinx-build and cleandocs target.
> As such:
> 
> - it won't run "make rustdoc";
> - no extra checks.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
> ---
>  .pylintrc                    |   2 +-
>  scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
>  2 files changed, 628 insertions(+), 1 deletion(-)
>  create mode 100755 scripts/sphinx-build-wrapper
> 

[...]

> diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
> new file mode 100755
> index 000000000000..5c728956b53c
> --- /dev/null
> +++ b/scripts/sphinx-build-wrapper
> @@ -0,0 +1,627 @@

[...]

> +    def handle_pdf(self, output_dirs):
> +        """
> +        Extra steps for PDF output.
> +
> +        As PDF is handled via a LaTeX output, after building the .tex file,
> +        a new build is needed to create the PDF output from the latex
> +        directory.
> +        """
> +        builds = {}
> +        max_len = 0
> +
> +        for from_dir in output_dirs:
> +            pdf_dir = os.path.join(from_dir, "../pdf")
> +            os.makedirs(pdf_dir, exist_ok=True)
> +
> +            if self.latexmk_cmd:
> +                latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> +            else:
> +                latex_cmd = [self.pdflatex]
> +
> +            latex_cmd.extend(shlex.split(self.latexopts))
> +
> +            tex_suffix = ".tex"
> +
> +            # Process each .tex file
> +            has_tex = False
> +            build_failed = False
> +            with os.scandir(from_dir) as it:
> +                for entry in it:
> +                    if not entry.name.endswith(tex_suffix):
> +                        continue
> +
> +                    name = entry.name[:-len(tex_suffix)]
> +                    has_tex = True
> +
> +                    try:
> +                        subprocess.run(latex_cmd + [entry.path],
> +                                       cwd=from_dir, check=True)

So, runs of latexmk (or xelatex) would be serialized, wouldn't they?
That would be a *huge* performance regression when I say:

    "make -j10 -O pdfdocs"

Current Makefile delegates .tex --> .pdf part of pdfdocs to sub make
of .../output/Makefile, which is generated on-the-fly by Sphinx's
latex builder.  That "-j10 -O" flag is passed to the sub make.

Another issue is that you are not deny-listing variable Noto CJK
fonts for latexmk/xelatex.  So this version of wrapper won't be able
to build translations.pdf if you have such variable fonts installed.
That failuer is not caught by your ad-hoc logic below.

> +                    except subprocess.CalledProcessError:
> +                        # LaTeX PDF error code is almost useless: it returns
> +                        # error codes even when build succeeds but has warnings.
> +                        pass
> +
> +                    # Instead of checking errors, let's do the next best thing:
> +                    # check if the PDF file was actually created.

I've seen cases where a corrupt .pdf file is left after premature crashes
of xdvipdfmx.  So, checking .pdf is not good enough for determining
success/failure.

One way to see if a .pdf file is properly formatted would be to run
"pdffonts" against the resulting .pdf.

For example, if I run "pdffonts translations.pdf" against the corrupted
one, I get:

   Syntax Error: Couldn't find trailer dictionary
   Syntax Error: Couldn't find trailer dictionary
   Syntax Error: Couldn't read xref table

, with the exit code of 1.
Against a successfully built translations.pdf, I get something like:

name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
JPRCQB+DejaVuSans-Bold               CID TrueType      Identity-H       yes yes yes      4  0
QFNXFP+DejaVuSerif-Bold              CID TrueType      Identity-H       yes yes yes     13  0
NMFBZR+NotoSerifCJKjp-Bold-Identity-H CID Type 0C       Identity-H       yes yes yes     15  0
WYMCYC+NotoSansCJKjp-Black-Identity-H CID Type 0C       Identity-H       yes yes yes     32  0
[...]

So I have to say this version of your wrapper does not look quite
ready to replace what you call "too much magic inside docs Makefile".

Regards,
Akira