linux-kernel - selftests: standardized results output?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <7F8EB1E5A1A71643ACAD0928E03FFDE90E3DABAC24@USMBX1.msg.corp.akamai.com>
Date:	Mon, 15 Dec 2014 11:42:12 -0500
From:	"Young, David" <dayoung@...mai.com>
To:	"linux-api@...r.kernel.org" <linux-api@...r.kernel.org>,
	"Shuah Khan (shuahkhan@...il.com)" <shuahkhan@...il.com>,
	"grant.likely@...retlab.ca" <grant.likely@...retlab.ca>
CC:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: selftests: standardized results output?

Hi,  I'm looking into what sorts of tools can consume the selftest output, and
found this on the wikipage:

https://kselftest.wiki.kernel.org/standardize_the_test_output

The current suggestion (as of the last-modified date on that wiki page for
October) is to use the Test Anything Protocol [TAP] for standard output.  I
notice that there is at least one test that conforms to TAP output, but
the majority of the tests are not using it.

There's also the kselftest.h file that suggests exit codes for the individual
test applications.

I'm interested in knowing what is the intention of this standardization, so
that I can put some work into a "glue" layer for a tool like buildbot,
autotest, or Jenkins for executing and consuming the results of these
selftests.

1) Is this output standard a "nice to have" that won't be much enforced?
2) Will the exit codes be utilized outside of the current makefile-based
   approach for executing the tests?  The current make target just runs all the
   tests without really concerning itself with the exit values of the
   individual tests.  It's simple, which isn't a bad thing, but it lacks a
   summarized result.  Is the intention to use a different harness to consume
   and report results?
3) Are the TAP results intended to be the exclusive (std)output of the tests,
   or will the tests report in a hybrid fashion?
   Such an example would be a test that produces some verbose stdout to the
   console, while simultaneously creating a TAP-compliant results.tap file as
   well... or vice versa with the stdout being TAP and a more verbose but
   free-form log.txt 

Thanks,

- David Young
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/