linux-kernel - Re: [PATCH] kunit: tool: improve compatibility of kunit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAFd5g45vKkpHhm0X3N+mRCAH8RNEXJTnFm2Dk9sPKzZY+qmMXQ@mail.gmail.com>
Date:   Fri, 20 Aug 2021 14:14:38 -0700
From:   Brendan Higgins <brendanhiggins@...gle.com>
To:     Rae Moar <rmoar@...gle.com>
Cc:     davidgow@...gle.com, dlatypov@...gle.com, shuah@...nel.org,
        kunit-dev@...glegroups.com, linux-kselftest@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: Re: [PATCH] kunit: tool: improve compatibility of kunit_parser with
 KTAP specification

On Fri, Aug 20, 2021 at 1:01 PM Rae Moar <rmoar@...gle.com> wrote:
>
> Update to kunit_parser to improve compatibility with KTAP
> specification including arbitrarily nested tests. Patch accomplishes
> three major changes:
>
> - Use a general Test object to represent all tests rather than TestCase
> and TestSuite objects. This allows for easier implementation of arbitrary
> levels of nested test and promotes the idea that both test suites and test
> cases are tests.
>
> - Print errors incrementally rather than all at once after the
> parsing finishes to maximize information given to the user in the
> case of the parser given invalid input and to increase the helpfulness
> of the timestamps given during printing.
>
> - Increase compatibility for different formats of input. Arbitrary levels
> of nested tests supported. Also, test cases and test suites are now
> supported to be present on the same level of testing.
>
> This patch now implements the KTAP specification as described here:
> https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/.
>
> This patch adjusts the kunit_tool_test.py file to check for
> the correct outputs from the new parser and adds a new test to check
> the parsing for a KTAP result log with correct format for multiple nested
> subtests (test_is_test_passed-all_passed_nested.log).
>
> This patch also alters the kunit_json.py file to allow for arbitrarily
> nested tests.
>
> Signed-off-by: Rae Moar <rmoar@...gle.com>

One minor question/potential issue below, otherwise:

Reviewed-by: Brendan Higgins <brendanhiggins@...gle.com>

[...]
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 75045aa0f8a1..ca760ee32096 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -106,10 +106,10 @@ class KUnitParserTest(unittest.TestCase):
>                 with open(log_path) as file:
>                         result = kunit_parser.extract_tap_lines(file.readlines())
>                 self.assertContains('TAP version 14', result)
> -               self.assertContains('   # Subtest: example', result)
> -               self.assertContains('   1..2', result)
> -               self.assertContains('   ok 1 - example_simple_test', result)
> -               self.assertContains('   ok 2 - example_mock_test', result)
> +               self.assertContains('# Subtest: example', result)
> +               self.assertContains('1..2', result)
> +               self.assertContains('ok 1 - example_simple_test', result)
> +               self.assertContains('ok 2 - example_mock_test', result)
>                 self.assertContains('ok 1 - example', result)

Do you have any test cases which accept "ok/not ok" lines without a
"-"? (Or tests for any other KTAP compliant Kselftest output?)

>         def test_output_with_prefix_isolated_correctly(self):

Cheers!