linux-kernel - Re: [RFC v2 PATCH 1/3] Documentation: add guidelines for writing testable code specifications

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+wEVJZS5GErRThB7wvQ5pDoVo_j=pKD4neN-U2qJEnTgHRYXg@mail.gmail.com>
Date: Tue, 21 Oct 2025 18:43:49 +0200
From: Gabriele Paoloni <gpaoloni@...hat.com>
To: David Hildenbrand <david@...hat.com>
Cc: Chuck Wolber <chuckwolber@...il.com>, Jonathan Corbet <corbet@....net>, shuah@...nel.org, 
	linux-kselftest@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-doc@...r.kernel.org, gregkh@...uxfoundation.org, linux-mm@...ck.org, 
	safety-architecture@...ts.elisa.tech, acarmina@...hat.com, 
	kstewart@...uxfoundation.org, chuck@...ber.net
Subject: Re: [RFC v2 PATCH 1/3] Documentation: add guidelines for writing
 testable code specifications

On Tue, Oct 21, 2025 at 6:34 PM David Hildenbrand <david@...hat.com> wrote:
>
> On 21.10.25 18:27, Gabriele Paoloni wrote:
> > Hi David
> >
> > On Tue, Oct 21, 2025 at 5:37 PM David Hildenbrand <david@...hat.com> wrote:
> >>
> >> On 20.10.25 23:02, Chuck Wolber wrote:
> >>> [Reposting with apologies for the dup and those inflicted by the broken Gmail
> >>> defaults. I have migrated away from Gmail, but some threads are still stuck
> >>> there.]
> >>>
> >>> On Mon, Oct 20, 2025 at 7:35 PM David Hildenbrand <david@...hat.com> wrote:
> >>>>
> >>>>>> +------------
> >>>>>> +The Documentation/doc-guide/kernel-doc.rst chapter describes how to document the code using the kernel-doc format, however it does not specify the criteria to be followed for writing testable specifications; i.e. specifications that can be used to for the semantic description of low level requirements.
> >>>>>
> >>>>> Please, for any future versions, stick to the 80-column limit; this is
> >>>>> especially important for text files that you want humans to read.
> >>>>>
> >>>>> As a nit, you don't need to start by saying what other documents don't
> >>>>> do, just describe the purpose of *this* document.
> >>>>>
> >>>>> More substantially ... I got a way into this document before realizing
> >>>>> that you were describing an addition to the format of kerneldoc
> >>>>> comments.  That would be good to make clear from the outset.
> >>>>>
> >>>>> What I still don't really understand is what is the *purpose* of this
> >>>>> formalized text?  What will be consuming it?  You're asking for a fair
> >>>>> amount of effort to write and maintain these descriptions; what's in it
> >>>>> for the people who do that work?
> >>>>
> >>>> I might be wrong, but sounds to me like someone intends to feed this to
> >>>> AI to generate tests or code.
> >>>
> >>> Absolutely not the intent. This is about the lossy process of converting human
> >>> ideas to code. Reliably going from code to test requires an understanding of
> >>> what was lost in translation. This project is about filling that gap.
> >>
> >> Thanks for clarifying. I rang my alarm bells too early :)
> >>
> >> I saw the LPC talk on this topic:
> >>
> >> https://lpc.events/event/19/contributions/2085/
> >>
> >> With things like "a test case can be derived from the testable
> >> expectation" one wonders how we get from the the doc to an actual test case.
> >
> > Probably it is the term derived that can be a bit misleading. The point is that
> > we need documented expectations that can be used to review and verify the
> > test cases against; so maybe better to say "a test case can be verified against
> > the testable expectation"
>
> On a high level (where we usually test with things like LTP) I would
> usually expect that the man pages properly describe the semantics of
> syscalls etc.

On a high level yes however there are two issues:
1) even the Posix standard define the behaviour of certain syscalls as
implementation specific
2) if all the details required to write testable specifications were maintained
as part of the manpage, these would become unmaintainable

For this reason specification must be broken down over the code in a
maintainable way

>
> That also feels like a better place to maintain such kind of information.
>
> Having that said, man-pages are frequently a bit outdated or imprecise
> .. or missing.
>
> Anyhow, I guess that will all be discussed in your LPC session I assume,
> I'll try to attend that one, thanks!

Sure Looking FWD to see you there

Gab

>
> --
> Cheers
>
> David / dhildenb
>