linux-kernel - Re: [PATCH] kunit: tool: fix --json output for skipped tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Sat, 9 Oct 2021 08:29:33 +0800
From:   David Gow <davidgow@...gle.com>
To:     Daniel Latypov <dlatypov@...gle.com>
Cc:     Brendan Higgins <brendanhiggins@...gle.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        KUnit Development <kunit-dev@...glegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH] kunit: tool: fix --json output for skipped tests

On Sat, Oct 9, 2021 at 7:24 AM Daniel Latypov <dlatypov@...gle.com> wrote:
>
> Currently, KUnit will report SKIPPED tests as having failed if one uses
> --json.
>
> Add the missing if statement to set the appropriate status ("SKIP").
> See https://api.kernelci.org/schema-test-case.html:
>   "status": {
>       "type": "string",
>       "description": "The status of the execution of this test case",
>       "enum": ["PASS", "FAIL", "SKIP", "ERROR"],
>       "default": "PASS"
>   },
> with this, we now can properly produce all four of the statuses.
>
> Fixes: 5acaf6031f53 ("kunit: tool: Support skipped tests in kunit_tool")
> Signed-off-by: Daniel Latypov <dlatypov@...gle.com>
> ---

Ah: thanks! I totally missed this!

Reviewed-by: David Gow <davidgow@...gle.com>

-- David

>  tools/testing/kunit/kunit_json.py      | 2 ++
>  tools/testing/kunit/kunit_tool_test.py | 6 ++++++
>  2 files changed, 8 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_json.py b/tools/testing/kunit/kunit_json.py
> index 746bec72b9ac..b6e66c5d64d1 100644
> --- a/tools/testing/kunit/kunit_json.py
> +++ b/tools/testing/kunit/kunit_json.py
> @@ -30,6 +30,8 @@ def _get_group_json(test: Test, def_config: str,
>                         test_case = {"name": subtest.name, "status": "FAIL"}
>                         if subtest.status == TestStatus.SUCCESS:
>                                 test_case["status"] = "PASS"
> +                       elif subtest.status == TestStatus.SKIPPED:
> +                               test_case["status"] = "SKIP"
>                         elif subtest.status == TestStatus.TEST_CRASHED:
>                                 test_case["status"] = "ERROR"
>                         test_cases.append(test_case)
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 77e61b0a40e8..b3cc0227843c 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -426,6 +426,12 @@ class KUnitJsonTest(unittest.TestCase):
>                         {'name': 'example_simple_test', 'status': 'ERROR'},
>                         result["sub_groups"][1]["test_cases"][0])
>
> +       def test_skipped_test_json(self):
> +               result = self._json_for('test_skip_tests.log')
> +               self.assertEqual(
> +                       {'name': 'example_skip_test', 'status': 'SKIP'},
> +                       result["sub_groups"][1]["test_cases"][1])
> +
>         def test_no_tests_json(self):
>                 result = self._json_for('test_is_test_passed-no_tests_run_with_header.log')
>                 self.assertEqual(0, len(result['sub_groups']))
>
> base-commit: 4a86e2973c31902a2a72f4f25f99b2367188ec5d
> --
> 2.33.0.882.g93a45727a2-goog
>