linux-kernel - [PATCH 0/2] selftests: tool: update format of kselftest TAP results to improve compatibility with KTAP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20210827225812.3247919-1-rmoar@google.com>
Date:   Fri, 27 Aug 2021 22:58:10 +0000
From:   Rae Moar <rmoar@...gle.com>
To:     brendanhiggins@...gle.com, davidgow@...gle.com,
        dlatypov@...gle.com, keescook@...omium.org, shuah@...nel.org
Cc:     kunit-dev@...glegroups.com, linux-kselftest@...r.kernel.org,
        linux-kernel@...r.kernel.org, Rae Moar <rmoar@...gle.com>
Subject: [PATCH 0/2] selftests: tool: update format of kselftest TAP results
 to improve compatibility with KTAP

This series of patches updates the format of kselftest TAP results to improve
compatibility with the proposed KTAP specification
(https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/).

Three changes:
- Change from "# " to "  " for indentation of nested tests
- Add subtest header line at start of tests with subtests. Line format
is "# Subtest: [name of test]".
- Remove TAP header in nested tests

Standardizing TAP results would not only allow for clearer documentation and ease of reading but by standardizing the format across different testing frameworks, we could also share the use of tools.

As an example:

This is a truncated version of TAP results from the kselftest ptrace with the new format changes:

TAP version 13
1..1
# selftests: ptrace: get_syscall_info
  # Subtest: selftests: ptrace: get_syscall_info
  1..1
  # Starting 1 tests from 1 test cases.
  #  RUN           global.get_syscall_info ...
  #            OK  global.get_syscall_info
  ok 1 global.get_syscall_info
  # PASSED: 1 / 1 tests passed.
  # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
ok 1 selftests: ptrace: get_syscall_info

With the new patch to update the KUnit parser to improve compatibility with the proposed KTAP specification, (https://lore.kernel.org/linux-kselftest/20210826195505.3066755-1-rmoar@google.com/) the above TAP results would be parsed as the following:

[20:46:09] ============================================================
[20:46:09] ===== selftests: ptrace: get_syscall_info (1 subtest) ======
[20:46:09] [PASSED] global.get_syscall_info
[20:46:09] ======= [PASSED] selftests: ptrace: get_syscall_info =======
[20:46:09] ============================================================
[20:46:09] Testing complete. Passed: 1, Failed: 0, Crashed: 0, Skipped: 0, Errors: 0

Thus, the kunit parser could become a useful tool for kselftest users.

Rae Moar (2):
  selftests: tool: Add subtest header line and change indentation format
    in TAP results
  Revert "selftests: Remove KSFT_TAP_LEVEL"

 tools/testing/selftests/Makefile            | 6 ++++++
 tools/testing/selftests/kselftest/prefix.pl | 2 +-
 tools/testing/selftests/kselftest/runner.sh | 7 ++++---
 3 files changed, 11 insertions(+), 4 deletions(-)

-- 
2.33.0.259.gc128427fd7-goog