linux-kernel - Re: [PATCH] selftests/clone3: Report descriptive test names

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <24cacac8-e67e-47e6-89b4-131d64de786b@linuxfoundation.org>
Date:   Wed, 18 Oct 2023 13:06:04 -0600
From:   Shuah Khan <skhan@...uxfoundation.org>
To:     Mark Brown <broonie@...nel.org>,
        Christian Brauner <brauner@...nel.org>,
        Shuah Khan <shuah@...nel.org>
Cc:     linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org,
        Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH] selftests/clone3: Report descriptive test names

On 10/17/23 17:01, Mark Brown wrote:
> The clone3() selftests currently report test results in a format that does
> not mesh entirely well with automation. They log output for each test such
> as:
> 
>    # [1382411] Trying clone3() with flags 0 (size 0)
>    # I am the parent (1382411). My child's pid is 1382412
>    # I am the child, my PID is 1382412
>    # [1382411] clone3() with flags says: 0 expected 0
>    ok 1 [1382411] Result (0) matches expectation (0)
> 
> This is not ideal for automated parsers since the text after the "ok 1" is
> treated as the test name when comparing runs by a lot of automation (tests
> routinely get renumbered due to things like new tests being added based on
> logical groupings). The PID means that the test names will frequently vary
> and the rest of the name being a description of results means several tests
> have identical text there.
> 
> Address this by refactoring things so that we have a static descriptive
> name for each test which we use when logging passes, failures and skips
> and since we now have a stable name for the test to hand log that before
> starting the test to address the common issue reading logs where the test
> name is only printed after any diagnostics. The result is:
> 
>   # Running test 'simple clone3()'
>   # [1562777] Trying clone3() with flags 0 (size 0)
>   # I am the parent (1562777). My child's pid is 1562778
>   # I am the child, my PID is 1562778
>   # [1562777] clone3() with flags says: 0 expected 0
>   ok 1 simple clone3()
> 
> In order to handle skips a bit more neatly this is done in a moderately
> invasive fashion where we move from a sequence of function calls to having
> an array of test parameters. This hopefully also makes it a little easier
> to see what the tests are doing when looking at both the source and the
> logs.
> 


Good change. Thank you.

Applied to linux-kselftest next for Linux 6.7-rc1.
> Signed-off-by: Mark Brown <broonie@...nel.org>
> ---
>   tools/testing/selftests/clone3/clone3.c | 265 +++++++++++++++++++++++---------
>   1 file changed, 192 insertions(+), 73 deletions(-)
> 

thanks,
-- Shuah