Message-ID: <20160701095258.68c45e13@recife.lan>
Date:	Fri, 1 Jul 2016 09:52:58 -0300
From:	Mauro Carvalho Chehab <mchehab@....samsung.com>
To:	Jani Nikula <jani.nikula@...el.com>
Cc:	Markus Heiser <markus.heiser@...marit.de>,
	Jonathan Corbet <corbet@....net>, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org, hverkuil@...all.nl,
	daniel.vetter@...ll.ch, airlied@...il.com,
	grant.likely@...retlab.ca, rdunlap@...radead.org, keithp@...thp.com
Subject: Re: [PATCH] doc: flat-table directive

Hi Jani,

Em Fri, 01 Jul 2016 13:44:17 +0300
Jani Nikula <jani.nikula@...el.com> escreveu:

> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@....samsung.com> wrote:
> > Not being able to compile just one docbook is a regression and breaks
> > my process. This needs to be fixed.  
> 
> Do you have a regression with *DocBook XML* on docs-next now? If yes,
> clearly that must be fixed ASAP.

No, but as we'll be switching from DocBook to Sphinx, it should be
possible for the maintainers to use the same or similar workflows.

The one requirement from myside, as the media maintainer, is to be
able to compile *just* the media Sphinx book. As I do it patch by
patch, and we usually merge ~500 patches per kernel version, the
build should be fast, and shouldn't be affected by any breakages
on other docs.

> As to Sphinx, I fear your DocBook workflows are so elaborate that we are
> bound to break some of them when switching over. Personally, I don't
> think that's all bad. 

Well, you're not the one that is responsible for fixing the media mess.
this workflow is there since 2006 at the V4L2 side, and has been improved
over the years. Thanks to it, our documentation always reflect the API
supported by the Kernel. At DVB, we didn't have anything like that, and
we had a *huge* gap between documentation and code. So, last year, we
fixed the gap and the DVB APIs that are more relevant are also in good
shape.

We should not go back to the dark ages, where gaps happened and nobody
detected, until several years after the code has been introduced, and
being hard to identify what the hack some obscure API bits really mean.

> It's an opportunity to make things better.

I'm all for making things better, but let's not make them worse.

> But we
> can't make all of that happen overnight for media. This is why we have
> Sphinx and DocBook side by side for now.

My plan is to swith over the media docbook in one Kernel cycle. Maintaining
two documentations side by side, with the high volume of changes they suffer
on each Kernel cycle is not feasible.

> We specifically wanted to make the switch for e.g. GPU documentation
> fast, act as guinea pigs, hit the issues, and make it easier for others
> in the end. We'd also love to get some of those validation things you
> have, for everyone, but using Sphinx instead of a bunch of scripts and
> sed on the input files.

Sphinx doesn't seem to be able to cross-reference the source code to
identify documentation gaps at the public APIs. So, we'll need an external
script to parse the code and get the API identifiers from the headers.

Yeah, the logic we use to parse the media headers could be more
generic and be used on other subsystems, and I'm all for it, but
semantic parsers are tricky. In order to make them generic, it would
very likely need to use a real lexical analyzer, like flex (or some
perl/python module that would increase the toolchain dependencies to build
the Kernel). Also, right now, the logic handles some special cases,
that are specific to the subsystem. By making it generic, we'll need
to either get rid of that or add per-book quirk configuration files.

> This is also one of the reasons why I so much want to keep everything
> behind one configuration file, and build everything in the Sphinx
> toolchain. To keep it all more uniform, to not duplicate stuff, and not
> deviate to some silos like we've done in the past. I think when we have
> things working, we can add dedicated config files for the select few
> things that have additional special needs. Media is probably one of
> them. But that said, I think we should be able to keep including that to
> the main documentation build too.

I agree with the general principle of building everything at the Sphinx
toolchain. But this requires a deep look, and it will take a while for
this goal to happen.

Yet, if you're talking about creating an unique huge document with
everything inside, I don't believe at an "one solution fits all" kind
of approach.

We have at least 3 different sets of documents right now:

set 1) Kernel-doc Kernel internal API documentation

This is actually split into 27 different DocBook templates. Among this set, 
3 of them documents also the uAPI: alsa-driver-api.tmpl, iio.tmpl and
networking.tmpl.

I guess it should be possible to group all of them altogether, as you
seem to want. Yet, I'm not sure if this is the best thing to do. What 
would be the advantage of grouping them? What would be the disadvantage?

Looking at the positive side, they all document APIs and they all share
about the same "DocumentationStyle", with is enforced via
kernel-doc-nano-HOWTO.txt. So, gluing them together won't look too much
like a Frankenstein monster.

set 2) "Pure" DocBooks, each of them organized as a separate book and that
don't use Kernel-doc:
	1. kernel-hacking.tmpl
	2. lsm.tmpl
	3. Media DocBook (media_api.tmpl, media-entities.tmpl, media-indices.tmpl)
	4. uio-howto.tmpl
	5. writing-an-alsa-driver.tmpl
	6. writing_musb_glue_layer.tmpl
	7. writing_usb_driver.tmpl

Each was written on a different time, by a different set of authors
and following their own documentation style. Some work has to be
done before being able to merge them, if ever desired, in order to make
them look as a hole. Yet, they deal different subjects. I suspect 
they'll be better alone than grouped into a single document.

Also, media is clearly big and complex enough to be a separate 
document, and has its own documentation style. Except perhaps
for ALSA, all the other 5 documents handle completely different
matters. I can't think why someone would want them grouped.

Also, I'm almost sure that, at least for what's currently at the
Media API DocBook, we'll need to be able to use a different CSS
template, because we have very long tables there, so we want to
use a different layout to use a "landscape" like style.

set 3) Pure text documents, under Documentation/.

In this set, there are also several different subjects, from very
technical stuff to things like CodingStyle and CodeOfConduct.

There are also documents there for the Kernel end-users, like:
kernel-parameters.txt. 

So, IMHO, we should either keep them alone, or create some
some sets like:
	- API documentation;
	- Userspace documentation;
	- Compilation documentation;
	- Kernel programming best practices.

Looking at the media documentation on set 3, there is a mix of updated
and really old docs under video4linux/ and dvb/. Those require extra
care: some could probably be trashed, others should be merged with the
media kernel-doc stuff. There are things there that won't fit on
either one of the categories, like video4linux/CARDLIST.*,
as they're meant to be a sort of end-users documentation.

In summary, grouping all those things into a single document
seems wrong.

Putting everything altoghether would be like gluing all your C
programming language books with your English books and dictionaries,
just because they all were found on the same library. Let's separate
the books into shelves, as it makes easier for the user to find
the right documentation he needs.

> Also, many thanks for digging into this and reporting issues that you do
> face.

Anytime. I'll keep looking into the issues I'm finding, as the end
goal for all of us is to have a better documented Kernel.

-- 
Thanks,
Mauro

Powered by blists - more mailing lists