linux-kernel - Re: [PATCH] selftests/mm: Fix test result reporting in gup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aCcvxaFc6DE_Mhr1@finisterre.sirena.org.uk>
Date: Fri, 16 May 2025 14:29:57 +0200
From: Mark Brown <broonie@...nel.org>
To: David Hildenbrand <david@...hat.com>
Cc: Andrew Morton <akpm@...ux-foundation.org>,
	Shuah Khan <shuah@...nel.org>, linux-mm@...ck.org,
	linux-kselftest@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] selftests/mm: Fix test result reporting in gup_longterm

On Fri, May 16, 2025 at 10:02:16AM +0200, David Hildenbrand wrote:
> On 15.05.25 10:57, Mark Brown wrote:

> > The kselftest framework uses the string logged when a test result is
> > reported as the unique identifier for a test, using it to track test
> > results between runs. The gup_longterm test completely fails to follow
> > this pattern, it runs a single test function repeatedly with various
> > parameters but each result report is a string logging an error message
> > which is fixed between runs.

> As the person who wrote that test (that you apparently didn't CC for some

I just CCed whoever get_maintainers told me to CC for the patch.

> reason), what exactly is the problem with that?

> We run tests. If all pass, we're happy, if one fails, we investigate.

None of the tooling is able to either distinguish between the multiple
tests that are being run in gup_longterm, nor compare the results of
multiple runs effectively.  If all the tests run they report themselves
as being duplicates of the same test name, if one of them starts failing
the effect is that one of the duplicates disappears and we get an
entirely new test that's never passed reported.  If multiple tests fail
it's even worse.  This means that UIs displaying test results end up
reporting things unclearly (Was there a regression or was a new test
that never worked added?  What was the test?).  Since it's difficult
to track the tests between runs tooling that does reporting of things
like "This last worked in X" in the UI doesn't work properly, and tool
driven bisection of issues similarly struggles since it can't tell
what's going on with any of the tests between runs.

Basically, the output is garbled and vastly less useful for people
running this as a matter of routine or as part of a broader kselftest
run.  For example with my own automation I probably won't notice that a
previously working test failed unless every single test fails, and newly
added tests that never worked are a much lower priority to the point
where I may never look at them depending on where they are.

If a selftest is reporting multiple tests it should report them with
names that are stable and unique.

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)