linux-kernel - Re: [PATCH v7 0/3] scanf: convert self-test to KUnit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAJ-ks9k9d4aX+P9F10h3TqHPOCHEQ5m=QyMAv7bU+Xyb3LRsOQ@mail.gmail.com>
Date: Wed, 12 Feb 2025 11:54:52 -0500
From: Tamir Duberstein <tamird@...il.com>
To: Andy Shevchenko <andriy.shevchenko@...ux.intel.com>
Cc: David Gow <davidgow@...gle.com>, Petr Mladek <pmladek@...e.com>, 
	Steven Rostedt <rostedt@...dmis.org>, Rasmus Villemoes <linux@...musvillemoes.dk>, 
	Sergey Senozhatsky <senozhatsky@...omium.org>, Andrew Morton <akpm@...ux-foundation.org>, 
	Shuah Khan <shuah@...nel.org>, Geert Uytterhoeven <geert@...ux-m68k.org>, linux-kernel@...r.kernel.org, 
	linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v7 0/3] scanf: convert self-test to KUnit

On Tue, Feb 11, 2025 at 12:26 PM Tamir Duberstein <tamird@...il.com> wrote:
>
> > Is it me who cut something or the above missing this information (total tests)?
> > If the latter, how are we supposed to answer to the question if the failed test
> > is from new bunch of cases I hypothetically added or regression of the existing
> > ones? Without this it seems like I need to go through all failures. OTOH it may
> > be needed anyway as failing test case needs an investigation.
>
> I assume you mean missing from the new output. Yeah, KUnit doesn't do
> this counting. Instead you get the test name in the failure message:
>
> > > > > > >     vsscanf("0 1e 3e43 31f0 0 0 5797 9c70", "%1hx %2hx %4hx %4hx %1hx %1hx %4hx %4hx", ...) expected 837828163 got 1044578334
> > > > > > >             not ok 1 " "
> > > > > > >         # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:92
>
> I think maybe you're saying: what if I add a new assertion (rather
> than a new test case), and I start getting failure reports - how do I
> know if the reporter is running old or new test code?
>
> In an ideal world the message above would give you all the information
> you need by including the line number from the test. This doesn't
> quite work out in this case because of the various test helper
> functions; you end up with a line number in the test helper rather
> than in the test itself. We could fix that by passing around __FILE__
> and __LINE__ (probably by wrapping the test helpers in a macro). What
> do you think?

I gave this a try locally, and it produced this output:

>     # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:94
> lib/scanf_kunit.c:555: vsscanf("0 1e 3e43 31f0 0 0 5797 9c70", "%1hx %2hx %4hx %4hx %1hx %1hx %4hx %4hx", ...) expected 837828163 got 1044578334
>         not ok 1 " "
>     # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:94
> lib/scanf_kunit.c:555: vsscanf("dc2:1c:0:3531:2621:5172:1:7", "%3hx:%2hx:%1hx:%4hx:%4hx:%4hx:%1hx:%1hx", ...) expected 892403712 got 28
>         not ok 2 ":"
>     # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:94
> lib/scanf_kunit.c:555: vsscanf("e083,8f6e,b,70ca,1,1,aab1,10e4", "%4hx,%4hx,%1hx,%4hx,%1hx,%1hx,%4hx,%4hx", ...) expected 1892286475 got 757614
>         not ok 3 ","
>     # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:94
> lib/scanf_kunit.c:555: vsscanf("2e72-8435-1-2fc-7cbd-c2f1-7158-2b41", "%4hx-%4hx-%1hx-%3hx-%4hx-%4hx-%4hx-%4hx", ...) expected 50069505 got 99381
>         not ok 4 "-"
>     # numbers_list_field_width_val_width: ASSERTION FAILED at lib/scanf_kunit.c:94
> lib/scanf_kunit.c:555: vsscanf("403/0/17/1/11e7/1/1fe8/34ba", "%3hx/%1hx/%2hx/%1hx/%4hx/%1hx/%4hx/%4hx", ...) expected 65559 got 1507328
>         not ok 5 "/"

Andy, Petr: what do you think? I've added this (and the original
output, as you requested) to the cover letter for when I reroll v8
(not before next week).