linux-kernel - Re: [RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as result code

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 01 Sep 2014 12:17:19 +0900
From:	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
To:	Shuah Khan <shuahkh@....samsung.com>
Cc:	Shuah Khan <shuah.kh@...sung.com>,
	Tom Zanussi <tom.zanussi@...ux.intel.com>,
	Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@...achi.com>,
	Oleg Nesterov <oleg@...hat.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	Namhyung Kim <namhyung@...nel.org>,
	Ingo Molnar <mingo@...nel.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [RFC PATCH v2 4/4] ftracetest: Add XFAIL/XPASS/UNSUPPORTED as
 result code

(2014/08/29 10:05), Shuah Khan wrote:
> On 08/26/2014 05:15 AM, Masami Hiramatsu wrote:
>> Add XFAIL, XPASS and UNSUPPORTED as a result code. These are
>> used for the results that test case is expected to fail or
>> unsupported feature (by config).
>> This also introduces PASS/FAIL/XFAIL/XPASS/UNSUP result codes
>> for each testcase. Since the results are not binary, each
>> testcase must use these code to return the test result.
>>
>> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
>> ---
>>  tools/testing/ftrace/ftracetest                    |   61 +++++++++++++++-----
>>  tools/testing/ftrace/test.d/basic1.tc              |    6 ++
>>  tools/testing/ftrace/test.d/basic2.tc              |    6 +-
>>  tools/testing/ftrace/test.d/basic3.tc              |    9 ++-
>>  .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   15 +++--
>>  tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   20 +++----
>>  tools/testing/ftrace/test.d/template               |    6 ++
>>  7 files changed, 84 insertions(+), 39 deletions(-)
>>
>> diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
>> index 0378c8a..bfcd56a 100755
>> --- a/tools/testing/ftrace/ftracetest
>> +++ b/tools/testing/ftrace/ftracetest
>> @@ -107,22 +107,53 @@ catlog() { #file
>>  }
>>  
>>  # Testcase management
>> +# Test result codes
>> +PASS=0	# The test succeeded.
>> +FAIL=1	# The test failed, but was expected to succeed.
>> +XFAIL=2	# The test failed, and was expected to fail.
>> +XPASS=3	# The test succeeded, but was expected to fail.
> 
> This one is confusing. It is still a failure. XFAIL case is
> expected to fail, using that convention XPASS should be
> expected to pass?

I see, but they have been already used as above by Dejagnu testing framework.
http://www.delorie.com/gnu/docs/dejagnu/dejagnu_6.html

Of course, it might be better to support only POSIX compatible results at
first.

> Can we use XFAIL0 for XFAIL pass case and XFAIL1 for XFAIL fail
> case?

No, new definition can confuse developers.
To avoid confusion, I'd like to use the same definitions as Dejegnu does.

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@...achi.com


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/