linux-kernel - Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250520150940.7d1cffec@sal.lan>
Date: Tue, 20 May 2025 15:09:51 +0200
From: Mauro Carvalho Chehab <mchehab+huawei@...nel.org>
To: Stephen Rothwell <sfr@...b.auug.org.au>, Nicolas Schier
 <nicolas.schier@...ux.dev>
Cc: Akira Yokosawa <akiyks@...il.com>, Randy Dunlap <rdunlap@...radead.org>,
 Jonathan Corbet <corbet@....net>, Andy Shevchenko
 <andriy.shevchenko@...ux.intel.com>, linux-doc@...r.kernel.org,
 linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 1/1] docs: kerneldoc.py: don't use Sphinx logger

Em Tue, 20 May 2025 21:39:06 +1000
Stephen Rothwell <sfr@...b.auug.org.au> escreveu:

> Hi Mauro,
> 
> I do appreciate your work on the document system.  Particularly you
> have made the process much faster.  However, see below.

Hi Stephen,

> On Tue, 20 May 2025 11:19:56 +0200 Mauro Carvalho Chehab <mchehab+huawei@...nel.org> wrote:
> >
> > If by "innocent typo" you mean removing/renaming/moving a file from
> > the Kernel tree without updating Makefiles, or adding them with wrong
> > names, then yes: I do think this should be fatal for the affected "make" 
> > targets. The quickest such build issues are caught, the best for everyone.  
> 
> They are found if you leave them as warnings and just continue on.  I
> have been finding them by when the current python version just crashes
> (without any useful message before your current change) I rerun the
> htmldocs using the perl version to get the name of the missing file and
> then report it so it can be fixed.  But the perl version continues on
> and produces all the other warnings so I can compare with the previous
> and only report new errors/warnigs.  Sometime these reports are not
> foxed for days or weeks (or months).
> 

The issues we're seeing weren't due to the conversion from perl to
python, but, instead, to the new logic that calls the Python methods
directly, instead of opening a shell to call kernel-doc.

Those caused some unintended changes, due to the way Sphinx works:

1. Sphinx has an overlay over the Python logger class. Such overlay
   has some logic to output colored messages, which is interesting.
   However, it also has a filtering logic which is affected by the
   flags passed when sphinx-build is called. That was actually
   suppressing not only verbose level logs, but also warnings and
   errors. The fix is a single-line change:

   -	kfiles = KernelFiles(out_style=out_style, logger=logger)
   +	kfiles = KernelFiles(out_style=out_style)

   The patch is at:
	https://lore.kernel.org/linux-doc/6b81b1aaa8446b4d850064dd38ffffa1a1cb6254.1747719873.git.mchehab+huawei@kernel.org/

2. Now that we're not opening a shell anymore, if kernel-doc.py
   dies, it could cause a fatal error on Sphinx. I sent today
   two patches fixing it:

   2.1 Don't stop Documentation/sphinx/kerneldoc.py if a crash
       happens:

	https://lore.kernel.org/linux-doc/064bac2f462c13f56154891d8f3fb788db94f325.1747730982.git.mchehab+huawei@kernel.org/

   2.2 Don't crash trying to retrieve kernel-doc messages from
       a file that doesn't exist. This is patch 1/2 on this
       series:
	https://lore.kernel.org/lkml/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

> So, please, now that you are producing more warnings, make a missing
> file just a warning (or at least continue on).  I have enough to do
> every day without having to run "make htmldocs" more than once.

OK. As I said, the crash was unintended. The idea was to preserve
the old behavior as much as possible. Yet, it could probably be a
good idea to have a way in the future to control the docs behavior
for the cases where non-existing files are detected, allowing it to
stop quickly on such cases, instead of taking several minutes to
generate an html ouptut, as this could be useful for CI automation.

In order to simplify for Jon, I'll send those tree altogether(*).

(*) I'll add a 4th patch, adding scripts/lib/kdoc to linux-doc
content. Without that, Lore doesn't store kernel-doc changes as
part of docs. See: 
	https://lore.kernel.org/linux-doc/d97e86c7176f671405b4c15d75cb951349022a23.1747730982.git.mchehab+huawei@kernel.org/

Regards,
Mauro