linux-kernel - Re: [PATCH] docs: Fix the docs build with Sphinx 6.0

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <e2d98136-3c4c-8910-d116-93070c48474d@gmail.com>
Date:   Tue, 10 Jan 2023 00:17:11 +0900
From:   Akira Yokosawa <akiyks@...il.com>
To:     Martin Liška <mliska@...e.cz>,
        Mauro Carvalho Chehab <mchehab@...nel.org>
Cc:     Jonathan Corbet <corbet@....net>, linux-doc@...r.kernel.org,
        linux-kernel@...r.kernel.org, Akira Yokosawa <akiyks@...il.com>
Subject: Re: [PATCH] docs: Fix the docs build with Sphinx 6.0

On Mon, 9 Jan 2023 15:14:46 +0100, Martin Liška wrote:
> On 1/8/23 15:01, Mauro Carvalho Chehab wrote:
>> Em Sat, 7 Jan 2023 14:17:24 +0900
>> Akira Yokosawa <akiyks@...il.com> escreveu:
>>
>>> On Wed, 04 Jan 2023 13:45:35 -0700, Jonathan Corbet wrote:
>>>> Sphinx 6.0 removed the execfile_() function, which we use as part of the
>>>> configuration process.  They *did* warn us...  Just open-code the
>>>> functionality as is done in Sphinx itself.
>>>>
>>>> Tested (using SPHINX_CONF, since this code is only executed with an
>>>> alternative config file) on various Sphinx versions from 2.5 through 6.0.
>>>>
>>>> Reported-by: Martin Liška <mliska@...e.cz>
>>>> Signed-off-by: Jonathan Corbet <corbet@....net>
>>>
>>> I have tested full builds of documentation with this change
>>> with Sphinx versions 1.7.9, 2.4.5, 3.4.3, 4.5.0, 5.3.0, and 6.0.0.
>>>
>>> Tested-by: Akira Yokosawa <akiyks@...il.com>
>>>
>>> That said, Sphinx 6.0.0 needs much more time and memory than earlier
>>> versions.
>>>
>>> FYI, I needed to limit parallel slot to 2 (make -j2) on a 16GB machine.
>>> If you are lucky, -j3 and -j4 might succeed. -j5 or more ended up in
>>> OOM situations for me:
>>>
>>> Comparison of elapsed time and maxresident with -j2:
>>>
>>>    ============== ============ ===========
>>>    Sphinx version elapsed time maxresident
>>>    ============== ============ ===========
>>>    5.3.0          10:16.81      937660
>>>    6.0.0          17:29.07     5292392
>>>    ============== ============ ===========
> 
> Hi.
> 
> I can confirm the regression, I bisected Sphinx revision that caused that
> and filled an upstream issues:
> https://github.com/sphinx-doc/sphinx/issues/11116

Thank you Martin for looking into this!

> 
> Cheers,
> Martin
> 
>>
>>  From the changelogs:
>>     https://www.sphinx-doc.org/en/master/changes.html
>>
>> It seems that 6.1 came with some performance optimizations, in particular:
>>
>>      Cache doctrees in the build environment during the writing phase.
>>
>>      Make all writing phase tasks support parallel execution.
>>
>>      Cache doctrees between the reading and writing phases.
>>
>> It would be nice if you could also test and check elapsed time
>> there too, as I suspect that 6.0 will have a very short usage, as
>> 6.1 was released just a few days after it.

Here is a table comparing 5.3.0, 6.0.1 and 6.1.2 taken on the same
machine (with 16GiB mem + 2GiB swap):

 ======  ===================================
                     elapsed time
         -----------------------------------
 Sphinx     -j1     -j2      -j4      -j6
 ======  ======== ======== ======== ========
  6.1.2  15:11.74 18:06.89 16:39.93      OOM
  6.0.1  15:28.19 17:22.15 16:31.30      OOM
  5.3.0  14:13.04 10:16.81  8:22.37  8:09.74
 ======  ======== ======== ======== ========

Note:
  - The -j1 run needs an explicit option given to sphinx-build by:
      make SPHINXOPTS="-q -j1" htmldocs
  - Once an OOM happens, -j4 runs are likely end up in OOM.

Looks like the non-parallel run is the cheapest option for mitigating
the regression.

        Thanks, Akira

>>
>> Regards,
>> Mauro.
>>
>>
>>
>>>
>>>          Thanks, Akira