linux-kernel - Re: [PATCH v1 9/9] selftests/mm: Run all tests from run

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5b4fcf62-98c3-458e-a0e7-8e86ec354cb9@sirena.org.uk>
Date:   Thu, 13 Jul 2023 16:43:29 +0100
From:   Mark Brown <broonie@...nel.org>
To:     Ryan Roberts <ryan.roberts@....com>
Cc:     David Hildenbrand <david@...hat.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Shuah Khan <shuah@...nel.org>,
        Jérôme Glisse <jglisse@...hat.com>,
        John Hubbard <jhubbard@...dia.com>,
        Florent Revest <revest@...omium.org>,
        "Liam R. Howlett" <Liam.Howlett@...cle.com>,
        linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v1 9/9] selftests/mm: Run all tests from run_vmtests.sh

On Thu, Jul 13, 2023 at 04:36:18PM +0100, Ryan Roberts wrote:
> On 13/07/2023 16:30, Mark Brown wrote:

> > The results parsers I'm aware of like the LAVA one will DTRT with nested
> > kselftests since that's required to pull see individual test cases run
> > by a single binary so it's the common case to see at least one level of
> > nesting.

> That's good to hear. But bear in mind that run_vmtests.sh does not use TAP. So
> you end up with a single top-level test who's result is reported with
> run_kselftest.sh's TAP output. Then you have a second level (run_vmtests.sh)
> using custom reporting, then _some_ of the tests invoked use TAP so you
> sometimes have TAP at level 3. But those tests at level 2 that don't do their
> own TAP output probably won't be parsed by LAVA?

I think that should mostly mean that all the tests that don't
individually produce KTAP output get ignored by parsers and those which
do produce KTAP output will be seen as nesting one level up from where
they are (ie, the individual cases will run directly from vmtest),
though there's likely to be confusion about expected run numbers for
things that actually pay attention to that.

> Since you agreed to put this into the CI, I was going to call this part "your
> problem" ;-)

It'll run, the results are a different story. :P

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)