linux-kernel - Re: [PATCH v9 6/6] scanf: further break kunit into test cases

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <Z9QtUoFdCoTXnkzE@pathway.suse.cz>
Date: Fri, 14 Mar 2025 14:21:22 +0100
From: Petr Mladek <pmladek@...e.com>
To: Tamir Duberstein <tamird@...il.com>
Cc: David Gow <davidgow@...gle.com>, Steven Rostedt <rostedt@...dmis.org>,
	Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
	Rasmus Villemoes <linux@...musvillemoes.dk>,
	Sergey Senozhatsky <senozhatsky@...omium.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Shuah Khan <shuah@...nel.org>,
	Geert Uytterhoeven <geert@...ux-m68k.org>,
	linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v9 6/6] scanf: further break kunit into test cases

On Fri 2025-03-07 06:27:39, Tamir Duberstein wrote:
> This gives us more granular test output.
> 
> Output before:
> 
> > KTAP version 1
> > 1..1
> >     KTAP version 1
> >     # Subtest: scanf
> >     # module: scanf_kunit
> >     1..10
> >     ok 1 numbers_simple
> >         KTAP version 1
> >         # Subtest: numbers_list
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list: pass:5 fail:0 skip:0 total:5
> >     ok 2 numbers_list
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_typemax
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_typemax: pass:5 fail:0 skip:0 total:5
> >     ok 3 numbers_list_field_width_typemax
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_width
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_val_width: pass:5 fail:0 skip:0 total:5
> >     ok 4 numbers_list_field_width_val_width
> >     ok 5 numbers_slice
> >     ok 6 numbers_prefix_overflow
> >     ok 7 test_simple_strtoull
> >     ok 8 test_simple_strtoll
> >     ok 9 test_simple_strtoul
> >     ok 10 test_simple_strtol
> > # scanf: pass:10 fail:0 skip:0 total:10
> > # Totals: pass:22 fail:0 skip:0 total:22
> > ok 1 scanf
> 
> Output after:
> 
> > KTAP version 1
> > 1..1
> >     KTAP version 1
> >     # Subtest: scanf
> >     # module: scanf_kunit
> >     1..21
> >     ok 1 numbers_simple
> >         KTAP version 1
> >         # Subtest: numbers_list_ll
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_ll: pass:5 fail:0 skip:0 total:5
> >     ok 2 numbers_list_ll
> >         KTAP version 1
> >         # Subtest: numbers_list_l
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_l: pass:5 fail:0 skip:0 total:5
> >     ok 3 numbers_list_l
> >         KTAP version 1
> >         # Subtest: numbers_list_d
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_d: pass:5 fail:0 skip:0 total:5
> >     ok 4 numbers_list_d
> >         KTAP version 1
> >         # Subtest: numbers_list_h
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_h: pass:5 fail:0 skip:0 total:5
> >     ok 5 numbers_list_h
> >         KTAP version 1
> >         # Subtest: numbers_list_hh
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_hh: pass:5 fail:0 skip:0 total:5
> >     ok 6 numbers_list_hh
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_ll
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_ll: pass:5 fail:0 skip:0 total:5
> >     ok 7 numbers_list_field_width_ll
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_l
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_l: pass:5 fail:0 skip:0 total:5
> >     ok 8 numbers_list_field_width_l
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_d
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_d: pass:5 fail:0 skip:0 total:5
> >     ok 9 numbers_list_field_width_d
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_h
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_h: pass:5 fail:0 skip:0 total:5
> >     ok 10 numbers_list_field_width_h
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_hh
> >         ok 1 " "
> >         ok 2 ":"
> >         ok 3 ","
> >         ok 4 "-"
> >         ok 5 "/"
> >     # numbers_list_field_width_hh: pass:5 fail:0 skip:0 total:5
> >     ok 11 numbers_list_field_width_hh
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_ll
> >         ok 1 ""
> >         ok 2 " "
> >         ok 3 ":"
> >         ok 4 ","
> >         ok 5 "-"
> >         ok 6 "/"
> >     # numbers_list_field_width_val_ll: pass:6 fail:0 skip:0 total:6
> >     ok 12 numbers_list_field_width_val_ll
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_l
> >         ok 1 ""
> >         ok 2 " "
> >         ok 3 ":"
> >         ok 4 ","
> >         ok 5 "-"
> >         ok 6 "/"
> >     # numbers_list_field_width_val_l: pass:6 fail:0 skip:0 total:6
> >     ok 13 numbers_list_field_width_val_l
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_d
> >         ok 1 ""
> >         ok 2 " "
> >         ok 3 ":"
> >         ok 4 ","
> >         ok 5 "-"
> >         ok 6 "/"
> >     # numbers_list_field_width_val_d: pass:6 fail:0 skip:0 total:6
> >     ok 14 numbers_list_field_width_val_d
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_h
> >         ok 1 ""
> >         ok 2 " "
> >         ok 3 ":"
> >         ok 4 ","
> >         ok 5 "-"
> >         ok 6 "/"
> >     # numbers_list_field_width_val_h: pass:6 fail:0 skip:0 total:6
> >     ok 15 numbers_list_field_width_val_h
> >         KTAP version 1
> >         # Subtest: numbers_list_field_width_val_hh
> >         ok 1 ""
> >         ok 2 " "
> >         ok 3 ":"
> >         ok 4 ","
> >         ok 5 "-"
> >         ok 6 "/"
> >     # numbers_list_field_width_val_hh: pass:6 fail:0 skip:0 total:6
> >     ok 16 numbers_list_field_width_val_hh
> >     ok 17 numbers_prefix_overflow
> >     ok 18 test_simple_strtoull
> >     ok 19 test_simple_strtoll
> >     ok 20 test_simple_strtoul
> >     ok 21 test_simple_strtol
> > # scanf: pass:21 fail:0 skip:0 total:21
> > # Totals: pass:86 fail:0 skip:0 total:86
> > ok 1 scanf
> 
> Signed-off-by: Tamir Duberstein <tamird@...il.com>

Thanks for trying this. But I personally do not find this as a big win.
It causes that the test log is longer than one screen and thus harder
to review.

It might be fine when running only this test and running it only once.
But I think that it is quite common to run more tests and then
it is easy to get lost.

I guess that there are tools for handling the KTAP format which might
make this easier but...

So, I would prefer to avoid this patch.

Best Regards,
Petr