linux-kernel - Re: [PATCH v1 4/5] selftests/nolibc: report: extrude the test status line

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20230709192652.97668-1-falcon@tinylab.org>
Date:   Mon, 10 Jul 2023 03:26:52 +0800
From:   Zhangjin Wu <falcon@...ylab.org>
To:     w@....eu
Cc:     arnd@...db.de, falcon@...ylab.org, linux-kernel@...r.kernel.org,
        linux-kselftest@...r.kernel.org, thomas@...ch.de
Subject: Re: [PATCH v1 4/5] selftests/nolibc: report: extrude the test status line

Hi, Willy

> On Thu, Jul 06, 2023 at 05:11:17PM +0800, Zhangjin Wu wrote:
> > two newlines are added around the test summary line to extrude the test
> > status.
> 
> But then we're back to making it annoying to check, having to figure
> if we need to grep -A or grep -B etc. With grep 'status:' we would get
> a synthetic status and the counters together. Why do you think it's
> not convenient ? Or am I the only one considering it useful to just
> run grep "status:" on all output files and figure a global status at
> once ?

Sorry, Willy, my commit message may mislead you a little.

The newlines are added around the whole test summary line (with the
status info), not only around the 'status info' ;-)

An example is added in our cover-letter (use '%3d' instead of '%03d'
here):

    ...
                                                 <-- newline here -->
    138 test(s): 135 passed,   2 skipped,   1 failed => status: failure
                                                 <-- newline here -->
    See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out

Or:

    ...
                                                 <-- newline here -->
    137 test(s): 134 passed,   3 skipped,   0 failed => status: warning
                                                 <-- newline here -->
    See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out

It is not for status grep, it is for developers to easily see the whole
summary line at a glance (I should add this in the commit message),
especially when we run tests for lots of architectures one by one
automatically, during the tests running, these newlines may help us to
see the status at a glance.

And further, if not consider pure-text, the colors may be more helpful,
for example, red for failed/failure, yellow for skipped/warning, green
for passed/success, for example:

    $ echo | awk 'END{printf("138 test(s): \033[32m135\033[0m passed, \033[33m  2\033[0m skipped, \033[31m  1\033[0m failed => status: \033[31mfailure\033[0m\n");}'
    138 test(s): 135 passed,   2 skipped,   1 failed => status: failure

But as we can see, the color control code is not readable and it may
break the simple "status: failure" grep, we should use something like
"status: .*failure" ;-)

It is possible to filter out the color info in the last run.out and only
reserve the colors info in the console.

    $ cat run.tmp.out | sed 's/\x1b\[[0-9;]*m//g' | col -bp > run.out

As a summary, the "status info grep" you proposed is very helpful to
summarize all of the tests after the testing finish, I do like it:

    $ grep "status: " /path/to/all/run.out
    138 test(s): 135 passed,   2 skipped,   1 failed => status: failure
    137 test(s): 134 passed,   3 skipped,   0 failed => status: warning

And these newlines (and even further with colors) are added to help
developers to see what happens during the tests running at a glance.

Thanks,
Zhangjin

> 
> Willy