linux-kernel - Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is unsupported tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20170704005232.a36903c24da6dad19027e12f@kernel.org>
Date:   Tue, 4 Jul 2017 00:52:32 +0900
From:   Masami Hiramatsu <mhiramat@...nel.org>
To:     Steven Rostedt <rostedt@...dmis.org>
Cc:     linux-kselftest@...r.kernel.org, shuah@...nel.org,
        Ingo Molnar <mingo@...hat.com>, linux-kernel@...r.kernel.org,
        naresh.kamboju@...aro.org
Subject: Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is
 unsupported tests

On Mon, 3 Jul 2017 09:15:43 -0400
Steven Rostedt <rostedt@...dmis.org> wrote:

> On Mon,  3 Jul 2017 14:25:36 +0900
> Masami Hiramatsu <mhiramat@...nel.org> wrote:
> 
> > Do not return failure exit code (1) for unsupported testcases,
> > since it is expected for stable kernels.
> > 
> > Previously, ftracetest is expected to run only on current
> > release for avoiding regressions. However, nowadays we run
> > it on stable kernels. This means some test cases must return
> > unsupported result. In such case, we should NOT exit
> > ftracetest with error status for unsupported results so that
> > kselftest (upper tests wrapper) shows it passed correctly.
> 
> I wonder if we should change kselftest instead. There are case where we
> want to report "unsupported" as a failure. For instance, I have tests
> where I enable everything, and if a test returns "unsupported" then it
> is a failure for me.

OK, that's a possible usecase. 

> 
> Can we add an option in kselftest, or to ftracetest that decides if
> unsupported is a failure or not? Otherwise I can not ack this patch.

I would rather like to add an option to ftracetest instead of
kselftest, because whether the tested feature should be supported
or not is hard to decide from testing framework. It should be
checked by manual.

Thank you,

> 
> -- Steve
> 
> 
> > 
> > Note that we continue to treat unresolved results as failure,
> > if test writers would like to notice user that the test result
> > should be reviewed, they can use exit_unresolved.
> > 
> > Signed-off-by: Masami Hiramatsu <mhiramat@...nel.org>
> > ---
> >  tools/testing/selftests/ftrace/ftracetest |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> > index 14a03ea..290cd42 100755
> > --- a/tools/testing/selftests/ftrace/ftracetest
> > +++ b/tools/testing/selftests/ftrace/ftracetest
> > @@ -187,7 +187,7 @@ eval_result() { # sigval
> >      $UNSUPPORTED)
> >        prlog "	[UNSUPPORTED]"
> >        UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
> > -      return 1 # this is not a bug, but the result should be reported.
> > +      return 0 # this is not a bug.
> >      ;;
> >      $XFAIL)
> >        prlog "	[XFAIL]"
> 


-- 
Masami Hiramatsu <mhiramat@...nel.org>