linux-kernel - Re: [PATCH/RFC] kunit/rtc: Add real support for very slow tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAMuHMdUqPQmX6KdtRK-mTuXmeGpzWNmbKoH9nEk54m60n-8aVg@mail.gmail.com>
Date: Fri, 28 Mar 2025 16:37:45 +0100
From: Geert Uytterhoeven <geert@...ux-m68k.org>
To: David Gow <davidgow@...gle.com>
Cc: Brendan Higgins <brendan.higgins@...ux.dev>, Rae Moar <rmoar@...gle.com>, 
	Alexandre Belloni <alexandre.belloni@...tlin.com>, linux-kselftest@...r.kernel.org, 
	kunit-dev@...glegroups.com, linux-rtc@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH/RFC] kunit/rtc: Add real support for very slow tests

Hi David,

On Fri, 28 Mar 2025 at 09:07, David Gow <davidgow@...gle.com> wrote:
> Thanks for sending this out: I think this raises some good questions
> about exactly how to handle long running tests (particularly on
> older/slower hardware).
>
> I've put a few notes below, but, tl;dr: I think these are all good
> changes, even if there's more we can do to better scale to slower
> hardware.
>
> On Fri, 28 Mar 2025 at 00:07, Geert Uytterhoeven <geert@...ux-m68k.org> wrote:
> >   2. Increase timeout by ten; ideally this should only be done for very
> >      slow tests, but I couldn't find how to access kunit_case.attr.case
> >      from kunit_try_catch_run(),
>
>
> My feeling for tests generally is:
> - Normal: effectively instant on modern hardware, O(seconds) on
> ancient hardware.
> - Slow: takes O(seconds) to run on modern hardware, O(minutes)..O(10s
> of minutes) on ancient hardware.
> - Very slow: O(minutes) or higher on modern hardware, infeasible on
> ancient hardware.
>
> Obviously the definition of "modern" and "ancient" hardware here is
> pretty arbitrary: I'm using "modern, high-end x86" ~4GHz as my
> "modern" example, and "66MHz 486" as my "ancient" one, but things like
> emulation or embedded systems fit in-between.
>
> Ultimately, I think the timeout probably needs to be configurable on a
> per-machine basis more than a per-test one, but having a 10x
> multiplier (or even a 100x multiplier) for very slow tests would also
> work for me.

Yes, adapting automatically to the speed of the target maachine
would be nice, but non-trivial.

> I quickly tried hacking together something to pass through the
> attribute and implement this. Diff (probably mangled by gmail) below:

[...]

Thanks!

> I'll get around to extending this to allow the "base timeout" to be
> configurable as a command-line option, too, if this seems like a good
> way to go.
>
> >   3. Mark rtc_time64_to_tm_test_date_range_1000 slow,
> >   4. Mark rtc_time64_to_tm_test_date_range_160000 very slow.
>
> Hmm... these are definitely fast enough on my "modern" machine that
> they probably only warrant "slow", not "very slow". But given they're
> definitely causing problems on older machines, I'm happy to go with
> marking the large ones very slow. (I've been waiting for them for
> about 45 minutes so far on my 486.)
>
> Do the time tests in kernel/time/time_test.c also need to be marked
> very slow, or does that run much faster on your setup?

Hmm, I did run time_test (insmod took (+7 minutes), but I don't
seem to have pass/fail output. Will rerun...

Indeed:

    # time64_to_tm_test_date_range.speed: slow

Another test that wanted to be marked as slow was:

    # kunit_platform_device_add_twice_fails_test: Test should be
marked slow (runtime: 30.788248702s)

I will rerun all, as it seems I have lost some logs...

> Is this causing you enough strife that you want it in as-is, straight
> away, or would you be happy with it being split up and polished a bit
> first -- particularly around supporting the more configurable timeout,
> and shifting the test changes into separate patches? (I'm happy to do
> that for you if you don't want to dig around in the somewhat messy
> KUnit try-catch stuff any further.)

This is definitely not something urgent for me.

Thanks!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@...ux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds