linux-kernel - Re: [PATCH v1] rtla: Set distinctive exit value for failed tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAP4=nvS7nAfrMfb5L8A4DU0ZtGY7cxVD9gTLL-xV9jW3rU1rTA@mail.gmail.com>
Date: Mon, 31 Mar 2025 08:30:20 +0200
From: Tomas Glozar <tglozar@...hat.com>
To: Costa Shulyupin <costa.shul@...hat.com>
Cc: Steven Rostedt <rostedt@...dmis.org>, John Kacur <jkacur@...hat.com>, 
	"Luis Claudio R. Goncalves" <lgoncalv@...hat.com>, Eder Zulian <ezulian@...hat.com>, 
	Dan Carpenter <dan.carpenter@...aro.org>, Jan Stancek <jstancek@...hat.com>, 
	linux-trace-kernel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v1] rtla: Set distinctive exit value for failed tests

Hi Costa,

Thanks for the patch, this is really useful, as we will be also able
to test whether rtla properly resets the threshold option (see below).

ne 30. 3. 2025 v 20:48 odesílatel Costa Shulyupin
<costa.shul@...hat.com> napsal:
> Failed tests return the same exit value as passed tests,
> requiring test frameworks to validate results by parsing output.

Please specify in the commit message more precisely what you mean by
"failed tests". That is, when running rtla with threshold, the exit
code is the same, regardless of whether the threshold is violated or
not.

> Set a unique exit value for failed tests to
> allow integration with testing frameworks without parsing output.

Note that this is not only for the test framework, users might use this as well.

Also, users might be already relying on rtla exiting with 0 on
threshold violation. What do you think about adding a new command line
argument, `--fail-on-threshold`, that would allow the user (including
the test suite) to trigger the new behavior?

>@@ -20,6 +20,6 @@ check "verify the --entries/-E param" \
> # and stopping on threshold.
> # If default period is not set, this will time out.
> check_with_osnoise_options "apply default period" \
>-       "osnoise hist -s 1" period_us=600000000
>+       "osnoise hist -s 1" 2 period_us=600000000

In the future, we can add regression tests like this:

check_with_osnoise_options "verify setting default threshold" \
        "osnoise hist -d 10" 0  stop_tracing_us=1

This will detect failure (return code 2) if the fix in commit
0122938a7a ("rtla: Always set all tracer options") no longer works.

Tomas