linux-kernel - Re: [RFC] A first shot at asciidoc-based formatted docs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <cover.1453809420.git.jani.nikula@intel.com>
Date:	Tue, 26 Jan 2016 14:08:45 +0200
From:	Jani Nikula <jani.nikula@...el.com>
To:	Jonathan Corbet <corbet@....net>, linux-doc@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org, Jani Nikula <jani.nikula@...el.com>,
	Daniel Vetter <daniel.vetter@...ll.ch>
Subject: Re: [RFC] A first shot at asciidoc-based formatted docs

On Tue, 26 Jan 2016, Jonathan Corbet <corbet@....net> wrote:
> So here is a proof-of-concept series showing how a fully asciidoc-based
> toolchain might work.  Lots of hackery here, this isn't meant to be applied
> to anything at this point, but it's a good start.  What this series has is:
>
>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>    doing this!  It was the kickstart that was needed to get this process
>    going.

Hooray! :)

>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>    ".adt", it's an asciidoc template; it's processed pretty much the same
>    way, except that kernel-doc gets the -asciidoc argument.
>
>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>    Naturally this was where most of the time got spent.  *Only* HTML output
>    works at the moment.
>
>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>    and the output is much pleasing, IMO.
>
> I'm sure there's a thousand details to deal with, and there is the issue of
> the other output formats.  asciidoctor claims to be able to create man
> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
> straightforward, so it should be possible to use xmlto as is done now.
>
> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
> anybody wants to mess with it.
>
> Comments?

I'm afraid we've done some overlapping work in the mean time, but I'm
happy we've both looked at the tool chain, and can have a more
meaningful conversation now.

I first took roughly the same approach as you did. I was really
impressed with the speed and the beauty of the produced HTML. The
trouble is, neither asciidoc nor asciidoctor can produce chunked (split
to several pages) HTML directly. This is a showstopper for the gpu
document which turns into 1.3 MB of HTML, which looks pretty but is a
paint to navigate. To do chunked output, you have to output DocBook and
handle that like we do now. So while I would like to have asciidoc
generate HTML directly for speed and beauty, I ended up going the
asciidoc to DocBook path. The upside is all the output formats are
supported.

(We could of course split the source documents, but then I believe we'd
have lots of trouble cross-referencing between the documents. I could be
proven wrong. I'd *like* to be proven wrong.)

One significant difference between our approaches is that I ditched
docproc out of the equation. Instead of having the docproc ! directives
in the asciidoc, I opted for using asciidoc's native include macro, with
specially crafted filename suffixes to specify what parts of the source
to include. Those files are then generated as intermediate asciidoc
files using kernel-doc, with dependencies set right. There's a bunch of
scripting involved, but it's pretty straight forward. 

So you'd have, for example,

include::drivers/gpu/drm/drm_ioctl.c,export[]

to include all the exported functions, or

include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]

to include the DOC: section. I think I'd like some version of this,
regardless of whether asciidoc generates HTML or DocBook. It lets make
parallelize all of kernel-doc processing for free, which is a big win.

Patches follow, also available in kernel-asciidoc branch of
git://people.freedesktop.org/~jani/drm (web interface
http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)

BR,
Jani.


Jani Nikula (10):
  kernel-doc: rewrite usage description, remove duplicated comments
  kernel-doc: add support for asciidoc output
  kernel-doc: support printing exported and non-exported symbols
  kernel-doc: add support for printing DOC: comments with escaped names
  scripts: add asciidoc-includes to extract includes from asciidoc
  scripts: add a kernel-doc helper for special invocation
  scripts: add tool for generating asciidoc dependencies and rules
  scripts: add a crude converter from DocBook tmpl to asciidoc
  Documentation: convert gpu.tmpl to gpu.txt
  Documentation: build asciidoc documentation

 Documentation/DocBook/Makefile |   37 +-
 Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
 Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
 scripts/asciidoc-includes      |    6 +
 scripts/kernel-doc             |  365 ++++-
 scripts/kernel-doc-deps        |   66 +
 scripts/kernel-doc-helper      |   70 +
 scripts/tmpl2asciidoc          |   39 +
 8 files changed, 1709 insertions(+), 3556 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt
 create mode 100755 scripts/asciidoc-includes
 create mode 100755 scripts/kernel-doc-deps
 create mode 100755 scripts/kernel-doc-helper
 create mode 100755 scripts/tmpl2asciidoc

-- 
2.1.4