linux-kernel - Re: [PATCH] selftests/cgroup: conform test to TAP format output

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJR+Y9LG+qrvDdbufNpBqW4+8oPBrmY3eABJzXBm9Y_DPN4XLg@mail.gmail.com>
Date: Fri, 14 Nov 2025 08:04:55 +0100
From: Sebastian Chlad <sebastian.chlad@...e.com>
To: Guopeng Zhang <zhangguopeng@...inos.cn>
Cc: mkoutny@...e.com, tj@...nel.org, hannes@...xchg.org, shuah@...nel.org, 
	cgroups@...r.kernel.org, linux-kselftest@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH] selftests/cgroup: conform test to TAP format output

On Fri, Nov 14, 2025 at 4:59 AM Guopeng Zhang <zhangguopeng@...inos.cn> wrote:
>
> Hi Michal,
>
> Thanks for reviewing and pointing out [1].
>
> > Could you please explain more why is the TAP layout beneficial?
> > (I understand selftest are for oneself, i.e. human readable only by default.)
>
> Actually, selftests are no longer just something for developers to view locally; they are now extensively
> run in CI and stable branch regression testing. Using a standardized layout means that general test runners
> and CI systems can parse the cgroup test results without any special handling.

I second that.
In fact, we do run some of those tests in the CI; i.e.
https://openqa.opensuse.org/tests/5453031#external
We added this: https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Parser/Format/KTAP.pm
to our CI
but frankly the use of the KTAP across the selftests is very
inconsistent, so we need to post-process some of the output files
quite a lot.
Therefore the more standardized the output, the better for any CI.

Small ask: should we amend the commit message to say KTAP?

That being said - the cgroups tests produce nice output which is easy
to parse and gives us no issues in our CI apart
from the shell tests, specifically test_cpuset_prs.sh.

We currently run the cgroup tests only internally because some of them
tend to fail when crossing resource-usage
boundaries and don’t provide clear information about by how much.
That ties into my earlier effort Michal linked here::
https://lore.kernel.org/all/rua6ubri67gh3b7atarbm5mggqgjyh6646mzkry2n2547jne4s@wvvpr3esi5es/

I’ll try to add the cgroup tests to the public openSUSE CI and will
test your patches.

>
> TAP provides a structured format that is both human-readable and machine-readable. The plan/result lines are parsed by tools,
> while the diagnostic lines can still contain human-readable debug information. Over time, other selftest suites (such as mm, KVM, mptcp, etc.)
> have also been converted to TAP-style output, so this change just brings the cgroup tests in line with that broader direction.
>
> > Or is this part of some tree-wide effort?
>
> This patch is not part of a formal, tree-wide conversion series I am running; it is an incremental step to align the
> cgroup C tests with the existing TAP usage. I started here because these tests already use ksft_test_result_*() and
> only require minor changes to generate proper TAP output.
>
> > I'm asking to better asses whether also the scripts listed in
> > Makefile:TEST_PROGS should be converted too.
>
> I agree that having them produce TAP output would benefit tooling and CI. I did not want to mix
> that into this change, but if you and other maintainers think this direction is reasonable,
> I would be happy to follow up and convert the cgroup shell tests to TAP as well.
>
> Thanks again for your review.
>
> Best regards,
> Guopeng
>
>