| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <5755CA35-2383-4EFA-B9CD-85573137F2A6@darmarit.de>
Date: Tue, 7 Jun 2016 11:14:16 +0200
From: Markus Heiser <markus.heiser@...marit.de>
To: Daniel Vetter <daniel.vetter@...ll.ch>
Cc: Jonathan Corbet <corbet@....net>,
Jani Nikula <jani.nikula@...el.com>,
Grant Likely <grant.likely@...retlab.ca>,
Mauro Carvalho Chehab <mchehab@....samsung.com>,
Keith Packard <keithp@...thp.com>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
linux-doc@...r.kernel.org, Hans Verkuil <hverkuil@...all.nl>
Subject: Re: [PATCH 0/7] add reST/sphinx-doc to linux documentation
Am 07.06.2016 um 09:54 schrieb Daniel Vetter <daniel.vetter@...ll.ch>:
> On Mon, Jun 6, 2016 at 6:32 PM, Markus Heiser <markus.heiser@...marit.de> wrote:
>> From: "Heiser, Markus" <markus.heiser@...marIT.de>
>>
>> Hi Jonathan, Jani, all -
>>
>> I merged the work from sphkerneldoc POC [1], into branch (on v4.7-rc2)
>>
>> git://github.com/return42/linux.git linux-doc-reST
>>
>> The specification of the implementation is the (new) kernel-doc-HOWTO book which
>> is a part of the patch series (also online avaliable at [2]).
>>
>> The kernel-doc-HOWTO book is a rewrite of the kernel-doc-nano-HOWTO.txt,
>> taking reST extension into account. It serves several purposes:
>>
>> * user guide for kernel developers
>> * specification of the kernel-doc parser
>> * test case
>>
>> The kernel_doc.py parser is a python implementation of the kernel-doc-HOWTO, the
>> kernel-doc perl script is not needed for reST builds. The python variant of the
>> kernel-doc parser supports the markup modes:
>>
>> * kernel-doc (vintage)
>> * reST
>>
>> This series also includes the reST directives describe in the kernel-doc-HOWTO:
>>
>> * kernel-doc (uses/imports the parser from kernel_doc.py)
>> * flat-table (table notation with meaningfull diffs)
>>
>> The Python/Sphinx extensions placed in the scripts/site-packages folder. With
>> this folder a minimal python infrastructure is provided.
>>
>> The basic sphinx-build infrastructure consists of:
>>
>> * Documentation/Makefile.reST & Documentation/conf.py
>>
>> Makefile and basic 'sphinx config' file to build the various reST
>> documents and output formats. Provides the basic sphinx-doc build
>> infrastructure including the *sphinx-subprojects* feature. With this
>> feature each book can be build and distributed stand-alone.
>>
>> * Documentation/books
>>
>> In this folder, the books with reST markup are placed. To
>> provide *sphinx-subprojects*, each book has its one folder and
>> a (optional) Documentation/books/{book-name}.conf file
>> which *overwrites* the basic configuration from Documentation/conf.py
>>
>> Any comments are welcome, but please read [2] first.
>
> I think getting the flat-table directive landed would be great. We
> have a similarly annoying table in gpu.tmpl where the ascii-art tables
> fall short and are annoying. We want to split it up, but that's not a
> short-term solution suitable for conversion. Atm it's licensed under
> gplv3, which is incompatible with the kernel, so need to fix that.
No problem, changed to gplv2
https://github.com/return42/linux/commit/97ee23e74384200fe79fe4557954a897725949e3
is this OK?
> I'm still not sold on the vintage-kerneldoc idea. At least in my
> experience (after first converting to asciidoc and now to sphinx) this
> is a non-issue. Yes, there's the oddball misrendering, but that's no
> worse than the oddball typo. And a really good way for
> drive-by-contributors and newbies to send in a few useful patches.
> Personally I'm totally happy to have that bit of ugliness, same way I
> don't reject patches just because they trip over checkpatch.pl in some
> minor detail. The downside otoh means we'll have to forever maintain 2
> versions of kernel-doc, and I don't want that.
This can be stopped by a reST lint process similar to what Jani mentioned.
> Jani&me have already
> ripped out some of the more bonghits features of kernel-doc (like
> anything with a \w: is a heading and other nonsense like that).
Yes, I removed it in the reST mode of the parser. This is also
a reason to differentiate between *vintage* and reST or you will
break tons of kernel-doc comments, which are valid in the term of
the *vintage* style.
> The only thing we really need to keep as special is the overall kerneldoc
> layout, and the special referencing using &, %, @ and friends.
Ok, I will try to add this highlighting feature to the reST mode.
I left it out because I feared that there are any collision with
reST markup, but with a closer look the fear might be unfounded.
> Reimplementing kernel-doc in python is also something Jani&I
> discussed. But I think it only makes sense if we go ahead and also
> modernize the parser, i.e. use something like python-clang to parse C,
> and build a modern parser-combinators thing for the kernel-doc itself.
> The regex state machinery is a horror show, whether it's written in
> cargo-culted perl or python.
Yes clang might be better solution and be more flexible on parsing
C, but it brings also much more decencies, but no one has tested
it against the kernel tree (I can't judge it, I have no experience
with) ... the regular expressions are proofed.
But anyway, if someone implements a C parser based on clang we can
use it, but before the python implementation is the best we have ;-)
> I agree that we need to update the kernel-doc howto (and Jani's
> working on that already too), but I think there's no need to split up
> that finely. Maybe that's a bit against sphinx best practices, but
> right now we have rather large documents (both plain text and
> docbook). Splitting definitely makes sense in some cases (e.g.
> gpu.tmpl is only this large because we couldn't do cross-template
> referencing), but by-and-large I think the current splitting is mostly
> reasonable.
You should be getting used to split the documents :-)
> A better demonstration vehicle is imo converting all
> existing .tmpl to .rst. And also making sure the new kernel-doc is
> still able to parse all existing kernel-doc comments, even when
> they're not pulled into .tmpl files.
This was my test case. Current migration of all .tmpl files
is based on v4.7-rc2 you will find the migrated reST books at [1]
and the rendered HTML at [2]
[1] https://github.com/return42/sphkerneldoc/tree/master/doc/books
[2] http://return42.github.io/sphkerneldoc/articles/books.html
It also finds the "Oops: Document generation inconsistency."
and mark it as to ToDo, e.g. see bottom of [3] or [4]
[3] http://return42.github.io/sphkerneldoc/books/gpu/index.html
[4] http://return42.github.io/sphkerneldoc/books/iio/index.html
> Jani has done that (using
> hacked-up versions of the sphinx-lint script) for his tree, making
> sure there's no unexpected changes anywhere.
> I think next steps would be:
> - rebase flat-table onto Jani's work and relicense under gplv2
> - look into rewriting kernel-doc in python more as a long-term project
I can't stop you ;-) .. but why not using my patch series. IMO at least
the python kernel-doc parser with deep and easy extendable
sphinx integration (compared to the pipe interface of the perl version),
the flat-table directive and the infrastructure with sphinx sub-projects
to compile and distribute each book separate are reasons to take it.
And yes, if you have any requirements, inform me, I will implement it.
Merging the highlighting feature to the reST mode ... seems the first ;-)
> - start converting docs instead - I really want to start reaping
> benefits of all this work as soon as possible.
As said, you can get all migrated .tmpl files from [1].
If you need something other migrated then v4.7-rc2 show me, where i can pull it,
I will send you a patch with migrated reST back. Or read [5]
[5] http://return42.github.io/sphkerneldoc/articles/dbtools.html
-- Markus --
>
> Thanks, Daniel
>
>
>>
>> With regards
>>
>> -- Markus --
>>
>> [1] https://return42.github.io/sphkerneldoc
>> [2] https://return42.github.io/sphkerneldoc/books/kernel-doc-HOWTO
>>
>>
>>
>> Heiser, Markus (7):
>> python: add scripts/site-packages
>> sphinx-doc: add basic sphinx-build infrastructure
>> kernel-doc-HOWTO: add kernel-doc specification
>> linuxdoc: add python package linuxdoc
>> kernel-doc parser: inital python implementation
>> kernel-doc directive: initial implementation
>> flat-table directive: initial implementation
>>
>> Documentation/.gitignore | 2 +
>> Documentation/Makefile.reST | 111 +
>> Documentation/books/.gitignore | 0
>> Documentation/books/kernel-doc-HOWTO.conf | 200 ++
>> .../books/kernel-doc-HOWTO/all-in-a-tumble-src.rst | 12 +
>> .../books/kernel-doc-HOWTO/all-in-a-tumble.h | 271 ++
>> .../books/kernel-doc-HOWTO/all-in-a-tumble.rst | 11 +
>> Documentation/books/kernel-doc-HOWTO/csv_table.txt | 6 +
>> Documentation/books/kernel-doc-HOWTO/index.rst | 46 +
>> .../kernel-doc-HOWTO/kernel-doc-components.rst | 35 +
>> .../kernel-doc-HOWTO/kernel-doc-directive.rst | 199 ++
>> .../books/kernel-doc-HOWTO/kernel-doc-examples.rst | 16 +
>> .../books/kernel-doc-HOWTO/kernel-doc-intro.rst | 85 +
>> .../books/kernel-doc-HOWTO/kernel-doc-syntax.rst | 171 ++
>> .../kernel-doc-HOWTO/reST-kernel-doc-mode.rst | 120 +
>> Documentation/books/kernel-doc-HOWTO/refs.txt | 15 +
>> .../books/kernel-doc-HOWTO/table-markup.rst | 499 ++++
>> .../books/kernel-doc-HOWTO/test/parser_test.h | 332 +++
>> .../kernel-doc-HOWTO/vintage-kernel-doc-mode.rst | 100 +
>> Documentation/conf.py | 357 +++
>> Documentation/sphinx-static/theme_overrides.css | 45 +
>> Documentation/sphinx-tex/Makefile | 88 +
>> scripts/site-python/.gitignore | 1 +
>> scripts/site-python/README | 14 +
>> scripts/site-python/linuxdoc/__init__.py | 8 +
>> scripts/site-python/linuxdoc/kernel_doc.py | 2764 ++++++++++++++++++++
>> scripts/site-python/linuxdoc/rstFlatTable.py | 356 +++
>> scripts/site-python/linuxdoc/rstKernelDoc.py | 369 +++
>> 28 files changed, 6233 insertions(+)
>> create mode 100644 Documentation/.gitignore
>> create mode 100644 Documentation/Makefile.reST
>> create mode 100644 Documentation/books/.gitignore
>> create mode 100644 Documentation/books/kernel-doc-HOWTO.conf
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble-src.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble.h
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/all-in-a-tumble.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/csv_table.txt
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/index.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-components.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-directive.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-examples.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-intro.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/kernel-doc-syntax.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/reST-kernel-doc-mode.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/refs.txt
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/table-markup.rst
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/test/parser_test.h
>> create mode 100644 Documentation/books/kernel-doc-HOWTO/vintage-kernel-doc-mode.rst
>> create mode 100644 Documentation/conf.py
>> create mode 100644 Documentation/sphinx-static/theme_overrides.css
>> create mode 100644 Documentation/sphinx-tex/Makefile
>> create mode 100644 scripts/site-python/.gitignore
>> create mode 100644 scripts/site-python/README
>> create mode 100644 scripts/site-python/linuxdoc/__init__.py
>> create mode 100755 scripts/site-python/linuxdoc/kernel_doc.py
>> create mode 100755 scripts/site-python/linuxdoc/rstFlatTable.py
>> create mode 100755 scripts/site-python/linuxdoc/rstKernelDoc.py
>>
>> --
>> v4.7-rc-2
>>
>
>
>
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Powered by blists - more mailing lists