lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Thu, 15 Feb 2024 09:38:08 -0800
From: Kees Cook <keescook@...omium.org>
To: Rae Moar <rmoar@...gle.com>
Cc: frowand.list@...il.com, davidgow@...gle.com, Tim.Bird@...y.com,
	shuah@...nel.org, brendanhiggins@...gle.com, dlatypov@...gle.com,
	tytso@...gle.com, gustavo.padovan@...labora.com,
	ricardo.canuelo@...labora.com, corbet@....net,
	kernelci@...ts.linux.dev, linux-kselftest@...r.kernel.org,
	kunit-dev@...glegroups.com, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: Re: [KTAP V2 PATCH v3] ktap_v2: add test metadata

On Thu, Feb 15, 2024 at 05:18:59PM +0000, Rae Moar wrote:
> Add specification for test metadata to the KTAP v2 spec.
> 
> KTAP v1 only specifies the output format of very basic test information:
> test result and test name. Any additional test information either gets
> added to general diagnostic data or is not included in the output at all.
> 
> The purpose of KTAP metadata is to create a framework to include and
> easily identify additional important test information in KTAP.
> 
> KTAP metadata could include any test information that is pertinent for
> user interaction before or after the running of the test. For example,
> the test file path or the test speed.
> 
> Since this includes a large variety of information, this specification
> will recognize notable types of KTAP metadata to ensure consistent format
> across test frameworks. See the full list of types in the specification.
> 
> Example of KTAP Metadata:
> 
>  KTAP version 2
>  #:ktap_test: main
>  #:ktap_arch: uml
>  1..1
>      KTAP version 2
>      #:ktap_test: suite_1
>      #:ktap_subsystem: example
>      #:ktap_test_file: lib/test.c
>      1..2
>      ok 1 test_1
>      #:ktap_test: test_2
>      #:ktap_speed: very_slow
>      # test_2 has begun
>      #:custom_is_flaky: true
>      ok 2 test_2
>  # suite_1 has passed
>  ok 1 suite_1
> 
> The changes to the KTAP specification outline the format, location, and
> different types of metadata.
> 
> Reviewed-by: David Gow <davidgow@...gle.com>
> Signed-off-by: Rae Moar <rmoar@...gle.com>

Thanks for the updates!

Reviewed-by: Kees Cook <keescook@...omium.org>

We'll need to update the test runners to adjust their outputs...

-- 
Kees Cook

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ