linux-kernel - Re: [PATCH v2 0/2] docs: Programmatically render MAINTAINERS into ReST

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20191002142413.09a12266@coco.lan>
Date:   Wed, 2 Oct 2019 14:24:13 -0300
From:   Mauro Carvalho Chehab <mchehab+samsung@...nel.org>
To:     Jonathan Corbet <corbet@....net>
Cc:     Kees Cook <keescook@...omium.org>,
        Jani Nikula <jani.nikula@...ux.intel.com>,
        linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2 0/2] docs: Programmatically render MAINTAINERS into
 ReST

Em Wed, 2 Oct 2019 10:25:35 -0600
Jonathan Corbet <corbet@....net> escreveu:

> On Tue,  1 Oct 2019 11:25:30 -0700
> Kees Cook <keescook@...omium.org> wrote:
> 
> > v1: https://lore.kernel.org/lkml/20190924230208.12414-1-keescook@chromium.org
> > 
> > v2: fix python2 utf-8 issue thanks to Jonathan Corbet
> > 
> > 
> > Commit log from Patch 2 repeated here for cover letter:
> > 
> > In order to have the MAINTAINERS file visible in the rendered ReST
> > output, this makes some small changes to the existing MAINTAINERS file
> > to allow for better machine processing, and adds a new Sphinx directive
> > "maintainers-include" to perform the rendering.  
> 
> OK, I've applied this.  Some notes, none of which really require any
> action...
> 
> It adds a new warning:
> 
>   /stuff/k/git/kernel/MAINTAINERS:40416: WARNING: unknown document: ../misc-devices/xilinx_sdfec           
> 
> I wonder if it's worth checking to be sure that documents referenced in
> MAINTAINERS actually exist.  OTOH, things as they are generate a warning,
> which is what we want anyway.

We check already links that start with Documentation/ and :doc: via the
scripts/documentation-file-ref-check script. It shouldn't probably be hard
to check other files at MAINTAINERS, although some care should be taken
for this to not take too long.

Right now, the script is very fast here (SSD), doing all the checks on
less than one second.

> I did various experiments corrupting the MAINTAINERS file and got some
> fairly unphotogenic results.  Again, though, I'm not sure that adding a
> bunch of code to generate warnings is really worth the trouble.
> 
> The resulting HTML file is 3.4MB and definitely makes my browser sweat when
> loading it :)

In the future, the script could split the MAINTAINERS output into multiple
files, for example:

	maintainers.rst (with an index file)
	maintainers_a.rst	# entries that starts with A
	maintainers_b.rst	# entries that starts with B
	...
	maintainers_z.rst	# entries that starts with Z

> 
> It adds about 20 seconds to a full "make htmldocs" build, which takes just
> over 3 minutes on the system in question.  So a 10% overhead, essentially.

Python is slow. On my tests with ABI automation, I also noticed some
performance degradation, and did some measurements. Reading all the
ABI files was very fast (specially since there I opted to use Perl,
with is a lot faster than Python). It was equally fast to output the data
(I wrote .rst files at the Python extension during my tests). What really
took almost all of the time of the extension was the Sphinx's internal
logic that parses ReST, which makes sense.

I guess that the increase of time for "make *docs" targets is
unavoidable: more documentation input means that it will take more time
to produce an output.

I suspect that, if we split the MAINTAINERS into multiple rst files,
the script will speedup, as Sphinx could better use the existing CPUs
in order to split its workload.

> 
> All told, it does what it's expected to do.  Thanks for doing this.
> 
> jon

Thanks,
Mauro