Message-ID: <54294C55.3030204@hitachi.com>
Date:	Mon, 29 Sep 2014 21:11:01 +0900
From:	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
To:	Steven Rostedt <rostedt@...dmis.org>
Cc:	Shuah Khan <shuah.kh@...sung.com>,
	Tom Zanussi <tom.zanussi@...ux.intel.com>,
	Oleg Nesterov <oleg@...hat.com>,
	Namhyung Kim <namhyung@...nel.org>,
	Fengguang Wu <fengguang.wu@...el.com>,
	Ingo Molnar <mingo@...nel.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH ftrace/for-next v6] ftracetest: Add POSIX.3 standard and
 XFAIL result codes

Steven, could you pick this instead of old v5 patch?

As you pointed, this version shows the log of "unsuppoted" test result.

Thank you,

(2014/09/29 21:02), Masami Hiramatsu wrote:
> Add XFAIL and POSIX 1003.3 standard codes (UNRESOLVED/
> UNTESTED/UNSUPPORTED) as result codes. These are used for the
> results that test case is expected to fail or unsupported
> feature (by config).
> 
> To return these result code, this introduces exit_unresolved,
> exit_untested, exit_unsupported and exit_xfail functions,
> which use real-time signals to notify the result code to
> ftracetest.
> 
> This also set "errexit" option for the testcases, so that
> the tests don't need to exit explicitly.
> 
> Note that if the test returns UNRESOLVED/UNSUPPORTED/FAIL,
> its test log including executed commands is shown on console
> and main logfile as below.
> 
>   ------
>   # ./ftracetest samples/
>   === Ftrace unit tests ===
>   [1] failure-case example        [FAIL]
>   execute: /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/fail.tc
>   + . /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/fail.tc
>   ++ cat non-exist-file
>   cat: non-exist-file: No such file or directory
>   [2] pass-case example   [PASS]
>   [3] unresolved-case example     [UNRESOLVED]
>   execute: /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/unresolved.tc
>   + . /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/unresolved.tc
>   ++ trap exit_unresolved INT
>   ++ kill -INT 29324
>   +++ exit_unresolved
>   +++ kill -s 38 29265
>   +++ exit 0
>   [4] unsupported-case example    [UNSUPPORTED]
>   execute: /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/unsupported.tc
>   + . /home/fedora/ksrc/linux-3/tools/testing/selftests/ftrace/samples/unsupported.tc
>   ++ exit_unsupported
>   ++ kill -s 40 29265
>   ++ exit 0
>   [5] untested-case example       [UNTESTED]
>   [6] xfail-case example  [XFAIL]
> 
>   # of passed:  1
>   # of failed:  1
>   # of unresolved:  1
>   # of untested:  1
>   # of unsupported:  1
>   # of xfailed:  1
>   # of undefined(test bug):  0
>   ------
> 
> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>
> Acked-by: Namhyung Kim <namhyung@...nel.org>
> 
> ---
> Changes in v6:
>  - Show the test log of UNSUPPORTED result.
>  - Check the test results and return 1 if any of them is failed.
> 
> Changes in v4:
>  - Remove XPASS and short-names.
>  - Use real-time signals for result code.
>  - Simplify test cases by using errexit option.
>  - Add sample scripts for result code.
> 
> Changes in v3:
>  - Add UNRESOLVED/UNTESTED codes.
>  - Fix to handle undefined codes.
>  - Add a document about return codes.
>  - Fix to show failure logs.
>  - Don't use -e option for echo since dash doesn't support it.
> ---
>  tools/testing/selftests/ftrace/README              |   37 ++++++
>  tools/testing/selftests/ftrace/ftracetest          |  124 ++++++++++++++++++--
>  tools/testing/selftests/ftrace/samples/fail.tc     |    4 +
>  tools/testing/selftests/ftrace/samples/pass.tc     |    3 
>  .../testing/selftests/ftrace/samples/unresolved.tc |    4 +
>  .../selftests/ftrace/samples/unsupported.tc        |    3 
>  tools/testing/selftests/ftrace/samples/untested.tc |    3 
>  tools/testing/selftests/ftrace/samples/xfail.tc    |    3 
>  .../selftests/ftrace/test.d/00basic/basic2.tc      |    3 
>  .../selftests/ftrace/test.d/00basic/basic3.tc      |    6 -
>  .../ftrace/test.d/kprobe/add_and_remove.tc         |   12 +-
>  .../selftests/ftrace/test.d/kprobe/busy_check.tc   |   15 +-
>  tools/testing/selftests/ftrace/test.d/template     |    5 +
>  13 files changed, 189 insertions(+), 33 deletions(-)
>  create mode 100644 tools/testing/selftests/ftrace/samples/fail.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/pass.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/unresolved.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/unsupported.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/untested.tc
>  create mode 100644 tools/testing/selftests/ftrace/samples/xfail.tc
> 
> diff --git a/tools/testing/selftests/ftrace/README b/tools/testing/selftests/ftrace/README
> index b8631f0..182e76f 100644
> --- a/tools/testing/selftests/ftrace/README
> +++ b/tools/testing/selftests/ftrace/README
> @@ -38,6 +38,43 @@ extension) and rewrite the test description line.
>   * The test cases should run on dash (busybox shell) for testing on
>     minimal cross-build environments.
>  
> + * Note that the tests are run with "set -e" (errexit) option. If any
> +   command fails, the test will be terminated immediately.
> +
> + * The tests can return some result codes instead of pass or fail by
> +   using exit_unresolved, exit_untested, exit_unsupported and exit_xfail.
> +
> +Result code
> +===========
> +
> +Ftracetest supports following result codes.
> +
> + * PASS: The test succeeded as expected. The test which exits with 0 is
> +         counted as passed test.
> +
> + * FAIL: The test failed, but was expected to succeed. The test which exits
> +         with !0 is counted as failed test.
> +
> + * UNRESOLVED: The test produced unclear or intermidiate results.
> +             for example, the test was interrupted
> +                       or the test depends on a previous test, which failed.
> +                       or the test was set up incorrectly
> +             The test which is in above situation, must call exit_unresolved.
> +
> + * UNTESTED: The test was not run, currently just a placeholder.
> +             In this case, the test must call exit_untested.
> +
> + * UNSUPPORTED: The test failed because of lack of feature.
> +               In this case, the test must call exit_unsupported.
> +
> + * XFAIL: The test failed, and was expected to fail.
> +          To return XFAIL, call exit_xfail from the test.
> +
> +There are some sample test scripts for result code under samples/.
> +You can also run samples as below:
> +
> +  # ./ftracetest samples/
> +
>  TODO
>  ====
>  
> diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> index 4c6c2fa..a8f81c7 100755
> --- a/tools/testing/selftests/ftrace/ftracetest
> +++ b/tools/testing/selftests/ftrace/ftracetest
> @@ -114,22 +114,106 @@ prlog "=== Ftrace unit tests ==="
>  
>  
>  # Testcase management
> +# Test result codes - Dejagnu extended code
> +PASS=0	# The test succeeded.
> +FAIL=1	# The test failed, but was expected to succeed.
> +UNRESOLVED=2  # The test produced indeterminate results. (e.g. interrupted)
> +UNTESTED=3    # The test was not run, currently just a placeholder.
> +UNSUPPORTED=4 # The test failed because of lack of feature.
> +XFAIL=5	# The test failed, and was expected to fail.
> +
> +# Accumulations
>  PASSED_CASES=
>  FAILED_CASES=
> +UNRESOLVED_CASES=
> +UNTESTED_CASES=
> +UNSUPPORTED_CASES=
> +XFAILED_CASES=
> +UNDEFINED_CASES=
> +TOTAL_RESULT=0
> +
>  CASENO=0
>  testcase() { # testfile
>    CASENO=$((CASENO+1))
>    prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
>  }
> -failed() {
> -  prlog "	[FAIL]"
> -  FAILED_CASES="$FAILED_CASES $CASENO"
> +
> +eval_result() { # retval sigval
> +  local retval=$2
> +  if [ $2 -eq 0 ]; then
> +    test $1 -ne 0 && retval=$FAIL
> +  fi
> +  case $retval in
> +    $PASS)
> +      prlog "	[PASS]"
> +      PASSED_CASES="$PASSED_CASES $CASENO"
> +      return 0
> +    ;;
> +    $FAIL)
> +      prlog "	[FAIL]"
> +      FAILED_CASES="$FAILED_CASES $CASENO"
> +      return 1 # this is a bug.
> +    ;;
> +    $UNRESOLVED)
> +      prlog "	[UNRESOLVED]"
> +      UNRESOLVED_CASES="$UNRESOLVED_CASES $CASENO"
> +      return 1 # this is a kind of bug.. something happened.
> +    ;;
> +    $UNTESTED)
> +      prlog "	[UNTESTED]"
> +      UNTESTED_CASES="$UNTESTED_CASES $CASENO"
> +      return 0
> +    ;;
> +    $UNSUPPORTED)
> +      prlog "	[UNSUPPORTED]"
> +      UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
> +      return 1 # this is not a bug, but the result should be reported.
> +    ;;
> +    $XFAIL)
> +      prlog "	[XFAIL]"
> +      XFAILED_CASES="$XFAILED_CASES $CASENO"
> +      return 0
> +    ;;
> +    *)
> +      prlog "	[UNDEFINED]"
> +      UNDEFINED_CASES="$UNDEFINED_CASES $CASENO"
> +      return 1 # this must be a test bug
> +    ;;
> +  esac
> +}
> +
> +# Signal handling for result codes
> +SIG_RESULT=
> +SIG_BASE=36	# Use realtime signals
> +SIG_PID=$$
> +
> +SIG_UNRESOLVED=$((SIG_BASE + UNRESOLVED))
> +exit_unresolved () {
> +  kill -s $SIG_UNRESOLVED $SIG_PID
> +  exit 0
> +}
> +trap 'SIG_RESULT=$UNRESOLVED' $SIG_UNRESOLVED
> +
> +SIG_UNTESTED=$((SIG_BASE + UNTESTED))
> +exit_untested () {
> +  kill -s $SIG_UNTESTED $SIG_PID
> +  exit 0
>  }
> -passed() {
> -  prlog "	[PASS]"
> -  PASSED_CASES="$PASSED_CASES $CASENO"
> +trap 'SIG_RESULT=$UNTESTED' $SIG_UNTESTED
> +
> +SIG_UNSUPPORTED=$((SIG_BASE + UNSUPPORTED))
> +exit_unsupported () {
> +  kill -s $SIG_UNSUPPORTED $SIG_PID
> +  exit 0
>  }
> +trap 'SIG_RESULT=$UNSUPPORTED' $SIG_UNSUPPORTED
>  
> +SIG_XFAIL=$((SIG_BASE + XFAIL))
> +exit_xfail () {
> +  kill -s $SIG_XFAIL $SIG_PID
> +  exit 0
> +}
> +trap 'SIG_RESULT=$XFAIL' $SIG_XFAIL
>  
>  # Run one test case
>  run_test() { # testfile
> @@ -137,14 +221,17 @@ run_test() { # testfile
>    local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
>    testcase $1
>    echo "execute: "$1 > $testlog
> -  (cd $TRACING_DIR; set -x ; . $1) >> $testlog 2>&1
> -  ret=$?
> -  if [ $ret -ne 0 ]; then
> -    failed
> -    catlog $testlog
> -  else
> -    passed
> +  SIG_RESULT=0
> +  # setup PID and PPID, $$ is not updated.
> +  (cd $TRACING_DIR; read PID _ < /proc/self/stat ;
> +   set -e; set -x; . $1) >> $testlog 2>&1
> +  eval_result $? $SIG_RESULT
> +  if [ $? -eq 0 ]; then
> +    # Remove test log if the test was done as it was expected.
>      [ $KEEP_LOG -eq 0 ] && rm $testlog
> +  else
> +    catlog $testlog
> +    TOTAL_RESULT=1
>    fi
>  }
>  
> @@ -152,8 +239,15 @@ run_test() { # testfile
>  for t in $TEST_CASES; do
>    run_test $t
>  done
> +
>  prlog ""
>  prlog "# of passed: " `echo $PASSED_CASES | wc -w`
>  prlog "# of failed: " `echo $FAILED_CASES | wc -w`
> -
> -test -z "$FAILED_CASES" # if no error, return 0
> +prlog "# of unresolved: " `echo $UNRESOLVED_CASES | wc -w`
> +prlog "# of untested: " `echo $UNTESTED_CASES | wc -w`
> +prlog "# of unsupported: " `echo $UNSUPPORTED_CASES | wc -w`
> +prlog "# of xfailed: " `echo $XFAILED_CASES | wc -w`
> +prlog "# of undefined(test bug): " `echo $UNDEFINED_CASES | wc -w`
> +
> +# if no error, return 0
> +exit $TOTAL_RESULT
> diff --git a/tools/testing/selftests/ftrace/samples/fail.tc b/tools/testing/selftests/ftrace/samples/fail.tc
> new file mode 100644
> index 0000000..15e35b9
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/fail.tc
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +# description: failure-case example
> +cat non-exist-file
> +echo "this is not executed"
> diff --git a/tools/testing/selftests/ftrace/samples/pass.tc b/tools/testing/selftests/ftrace/samples/pass.tc
> new file mode 100644
> index 0000000..d015493
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/pass.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: pass-case example
> +return 0
> diff --git a/tools/testing/selftests/ftrace/samples/unresolved.tc b/tools/testing/selftests/ftrace/samples/unresolved.tc
> new file mode 100644
> index 0000000..41e99d3
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/unresolved.tc
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +# description: unresolved-case example
> +trap exit_unresolved INT
> +kill -INT $PID
> diff --git a/tools/testing/selftests/ftrace/samples/unsupported.tc b/tools/testing/selftests/ftrace/samples/unsupported.tc
> new file mode 100644
> index 0000000..45910ff
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/unsupported.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: unsupported-case example
> +exit_unsupported
> diff --git a/tools/testing/selftests/ftrace/samples/untested.tc b/tools/testing/selftests/ftrace/samples/untested.tc
> new file mode 100644
> index 0000000..35a4594
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/untested.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: untested-case example
> +exit_untested
> diff --git a/tools/testing/selftests/ftrace/samples/xfail.tc b/tools/testing/selftests/ftrace/samples/xfail.tc
> new file mode 100644
> index 0000000..9dd3953
> --- /dev/null
> +++ b/tools/testing/selftests/ftrace/samples/xfail.tc
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: xfail-case example
> +cat non-exist-file || exit_xfail
> diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> index b04f30d..bf9a7b0 100644
> --- a/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> +++ b/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
> @@ -1,6 +1,7 @@
>  #!/bin/sh
>  # description: Basic test for tracers
> +test -f available_tracers
>  for t in `cat available_tracers`; do
> -  echo $t > current_tracer || exit 1
> +  echo $t > current_tracer
>  done
>  echo nop > current_tracer
> diff --git a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> index 0c1a3a2..bde6625 100644
> --- a/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> +++ b/tools/testing/selftests/ftrace/test.d/00basic/basic3.tc
> @@ -1,8 +1,8 @@
>  #!/bin/sh
>  # description: Basic trace clock test
> -[ -f trace_clock ] || exit 1
> +test -f trace_clock
>  for c in `cat trace_clock | tr  -d \[\]`; do
> -  echo $c > trace_clock || exit 1
> -  grep '\['$c'\]' trace_clock || exit 1
> +  echo $c > trace_clock
> +  grep '\['$c'\]' trace_clock
>  done
>  echo local > trace_clock
> diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> index 5ddfb47..1b8b665 100644
> --- a/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> +++ b/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> @@ -1,11 +1,11 @@
>  #!/bin/sh
>  # description: Kprobe dynamic event - adding and removing
>  
> -[ -f kprobe_events ] || exit 1
> +[ -f kprobe_events ] || exit_unsupported # this is configurable
>  
> -echo 0 > events/enable || exit 1
> -echo > kprobe_events || exit 1
> -echo p:myevent do_fork > kprobe_events || exit 1
> -grep myevent kprobe_events || exit 1
> -[ -d events/kprobes/myevent ] || exit 1
> +echo 0 > events/enable
> +echo > kprobe_events
> +echo p:myevent do_fork > kprobe_events
> +grep myevent kprobe_events
> +test -d events/kprobes/myevent
>  echo > kprobe_events
> diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> index 588fde97..b55c840 100644
> --- a/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> +++ b/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> @@ -1,14 +1,13 @@
>  #!/bin/sh
>  # description: Kprobe dynamic event - busy event check
>  
> -[ -f kprobe_events ] || exit 1
> +[ -f kprobe_events ] || exit_unsupported
>  
> -echo 0 > events/enable || exit 1
> -echo > kprobe_events || exit 1
> -echo p:myevent do_fork > kprobe_events || exit 1
> -[ -d events/kprobes/myevent ] || exit 1
> -echo 1 > events/kprobes/myevent/enable || exit 1
> +echo 0 > events/enable
> +echo > kprobe_events
> +echo p:myevent do_fork > kprobe_events
> +test -d events/kprobes/myevent
> +echo 1 > events/kprobes/myevent/enable
>  echo > kprobe_events && exit 1 # this must fail
> -echo 0 > events/kprobes/myevent/enable || exit 1
> +echo 0 > events/kprobes/myevent/enable
>  echo > kprobe_events # this must succeed
> -
> diff --git a/tools/testing/selftests/ftrace/test.d/template b/tools/testing/selftests/ftrace/test.d/template
> index ce5f735..5448f7a 100644
> --- a/tools/testing/selftests/ftrace/test.d/template
> +++ b/tools/testing/selftests/ftrace/test.d/template
> @@ -1,4 +1,9 @@
>  #!/bin/sh
>  # description: %HERE DESCRIBE WHAT THIS DOES%
>  # you have to add ".tc" extention for your testcase file
> +# Note that all tests are run with "errexit" option.
> +
>  exit 0 # Return 0 if the test is passed, otherwise return !0
> +# If the test could not run because of lack of feature, call exit_unsupported
> +# If the test returned unclear results, call exit_unresolved
> +# If the test is a dummy, or a placeholder, call exit_untested
> 
> 
> 


-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@...achi.com


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists