Message-ID: <CAMDBHY+AoF3_2ovczfyYYDQZF=b4rrNQ=xo6EO7d5-HtOjymFw@mail.gmail.com>
Date:   Wed, 14 Feb 2018 17:46:02 -0500
From:   Lucas Bates <lucasb@...atatu.com>
To:     "Brenda J. Butler" <bjb@...atatu.com>
Cc:     davem@...emloft.net, kernel@...atatu.com,
        Cong Wang <xiyou.wangcong@...il.com>,
        Jiri Pirko <jiri@...nulli.us>, Chris Mi <chrism@...lanox.com>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: Re: [PATCH net-next 7/7] tools: tc-testing: Update README and TODO

On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler <bjb@...atatu.com> wrote:
> Signed-off-by: Brenda J. Butler <bjb@...atatu.com>

Acked-by: Lucas Bates <lucasb@...atatu.com>
> ---
>  tools/testing/selftests/tc-testing/README   | 173 +++++++++++++++++++++++++---
>  tools/testing/selftests/tc-testing/TODO.txt |  25 +++-
>  2 files changed, 179 insertions(+), 19 deletions(-)
>
> diff --git a/tools/testing/selftests/tc-testing/README b/tools/testing/selftests/tc-testing/README
> index 970ff294fec8..3a0336782d2d 100644
> --- a/tools/testing/selftests/tc-testing/README
> +++ b/tools/testing/selftests/tc-testing/README
> @@ -14,11 +14,11 @@ REQUIREMENTS
>
>  *  The kernel must have network namespace support
>
> -*   The kernel must have veth support available, as a veth pair is created
> +*  The kernel must have veth support available, as a veth pair is created
>     prior to running the tests.
>
> -*  All tc-related features must be built in or available as modules.
> -   To check what is required in current setup run:
> +*  All tc-related features being tested must be built in or available as
> +   modules.  To check what is required in current setup run:
>     ./tdc.py -c
>
>     Note:
> @@ -44,10 +44,13 @@ using the -p option when running tdc:
>  RUNNING TDC
>  -----------
>
> -To use tdc, root privileges are required. tdc will not run otherwise.
> +To use tdc, root privileges are required.  This is because the
> +commands being tested must be run as root.  The code that enforces
> +execution by root uid has been moved into a plugin (see PLUGIN
> +ARCHITECTURE, below).
>
> -All tests are executed inside a network namespace to prevent conflicts
> -within the host.
> +If nsPlugin is linked, all tests are executed inside a network
> +namespace to prevent conflicts within the host.
>
>  Running tdc without any arguments will run all tests. Refer to the section
>  on command line arguments for more information, or run:
> @@ -59,6 +62,33 @@ output captured from the failing test will be printed immediately following
>  the failed test in the TAP output.
>
>
> +OVERVIEW OF TDC EXECUTION
> +-------------------------
> +
> +One run of tests is considered a "test suite" (this will be refined in the
> +future).  A test suite has one or more test cases in it.
> +
> +A test case has four stages:
> +
> +  - setup
> +  - execute
> +  - verify
> +  - teardown
> +
> +The setup and teardown stages can run zero or more commands.  The setup
> +stage does some setup if the test needs it.  The teardown stage undoes
> +the setup and returns the system to a "neutral" state so any other test
> +can be run next.  These two stages require any commands run to return
> +success, but do not otherwise verify the results.
> +
> +The execute and verify stages each run one command.  The execute stage
> +tests the return code against one or more acceptable values.  The
> +verify stage checks the return code for success, and also compares
> +the stdout with a regular expression.
> +
> +Each of the commands in any stage will run in a shell instance.
> +
> +
>  USER-DEFINED CONSTANTS
>  ----------------------
>
> @@ -70,23 +100,132 @@ executed as part of the test. More will be added as test cases require.
>  Example:
>         $TC qdisc add dev $DEV1 ingress
>
> +The NAMES values are used to substitute into the commands in the test cases.
> +
>
>  COMMAND LINE ARGUMENTS
>  ----------------------
>
>  Run tdc.py -h to see the full list of available arguments.
>
> --p PATH           Specify the tc executable located at PATH to be used on this
> -                  test run
> --c                Show the available test case categories in this test file
> --c CATEGORY       Run only tests that belong to CATEGORY
> --f FILE           Read test cases from the JSON file named FILE
> --l [CATEGORY]     List all test cases in the JSON file. If CATEGORY is
> -                  specified, list test cases matching that category.
> --s ID             Show the test case matching ID
> --e ID             Execute the test case identified by ID
> --i                Generate unique ID numbers for test cases with no existing
> -                  ID number
> +usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
> +              [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]
> +              [-d DEVICE] [-n NS] [-V]
> +
> +Linux TC unit tests
> +
> +optional arguments:
> +  -h, --help            show this help message and exit
> +  -p PATH, --path PATH  The full path to the tc executable to use
> +  -v, --verbose         Show the commands that are being run
> +  -d DEVICE, --device DEVICE
> +                        Execute the test case in flower category
> +
> +selection:
> +  select which test cases: files plus directories; filtered by categories
> +  plus testids
> +
> +  -D DIR [DIR ...], --directory DIR [DIR ...]
> +                        Collect tests from the specified directory(ies)
> +                        (default [tc-tests])
> +  -f FILE [FILE ...], --file FILE [FILE ...]
> +                        Run tests from the specified file(s)
> +  -c [CATG [CATG ...]], --category [CATG [CATG ...]]
> +                        Run tests only from the specified category/ies, or if
> +                        no category/ies is/are specified, list known
> +                        categories.
> +  -e ID [ID ...], --execute ID [ID ...]
> +                        Execute the specified test cases with specified IDs
> +
> +action:
> +  select action to perform on selected test cases
> +
> +  -l, --list            List all test cases, or those only within the
> +                        specified category
> +  -s, --show            Display the selected test cases
> +  -i, --id              Generate ID numbers for new test cases
> +
> +netns:
> +  options for nsPlugin(run commands in net namespace)
> +
> +  -n NS, --namespace NS
> +                        Run commands in namespace NS
> +
> +valgrind:
> +  options for valgrindPlugin (run command under test under Valgrind)
> +
> +  -V, --valgrind        Run commands under valgrind
> +
> +
> +PLUGIN ARCHITECTURE
> +-------------------
> +
> +There is now a plugin architecture, and some of the functionality that
> +was in the tdc.py script has been moved into the plugins.
> +
> +The plugins are in the directory plugin-lib.  The are executed from
> +directory plugins.  Put symbolic links from plugins to plugin-lib,
> +and name them according to the order you want them to run.
> +
> +Example:
> +
> +bjb@bee:~/work/tc-testing$ ls -l plugins
> +total 4
> +lrwxrwxrwx  1 bjb  bjb    27 Oct  4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
> +lrwxrwxrwx  1 bjb  bjb    25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
> +-rwxr-xr-x  1 bjb  bjb     0 Sep 29 15:56 __init__.py
> +
> +The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
> +must be called "SubPlugin" so tdc can find them.  They are
> +distinguished from each other in the python program by their module
> +name.
> +
> +This base class supplies "hooks" to run extra functions.  These hooks are as follows:
> +
> +pre- and post-suite
> +pre- and post-case
> +pre- and post-execute stage
> +adjust-command (runs in all stages and receives the stage name)
> +
> +The pre-suite hook receives the number of tests and an array of test ids.
> +This allows you to dump out the list of skipped tests in the event of a
> +failure during setup or teardown stage.
> +
> +The pre-case hook receives the ordinal number and test id of the current test.
> +
> +The adjust-command hook receives the stage id (see list below) and the
> +full command to be executed.  This allows for last-minute adjustment
> +of the command.
> +
> +The stages are identified by the following strings:
> +
> +  - pre  (pre-suite)
> +  - setup
> +  - command
> +  - verify
> +  - teardown
> +  - post (post-suite)
> +
> +
> +To write a plugin, you need to inherit from TdcPlugin in
> +TdcPlugin.py.  To use the plugin, you have to put the
> +implementation file in plugin-lib, and add a symbolic link to it from
> +plugins.  It will be detected at run time and invoked at the
> +appropriate times.  There are a few examples in the plugin-lib
> +directory:
> +
> +  - rootPlugin.py:
> +      implements the enforcement of running as root
> +  - nsPlugin.py:
> +      sets up a network namespace and runs all commands in that namespace
> +  - valgrindPlugin.py
> +      runs each command in the execute stage under valgrind,
> +      and checks for leaks.
> +      This plugin will output an extra test for each test in the test file,
> +      one is the existing output as to whether the test passed or failed,
> +      and the other is a test whether the command leaked memory or not.
> +      (This one is a preliminary version, it may not work quite right yet,
> +      but the overall template is there and it should only need tweaks.)
>
>
>  ACKNOWLEDGEMENTS
> diff --git a/tools/testing/selftests/tc-testing/TODO.txt b/tools/testing/selftests/tc-testing/TODO.txt
> index 6a266d811a78..c40698557e2f 100644
> --- a/tools/testing/selftests/tc-testing/TODO.txt
> +++ b/tools/testing/selftests/tc-testing/TODO.txt
> @@ -5,6 +5,27 @@ tc Testing Suite To-Do list:
>
>  - Add support for multiple versions of tc to run successively
>
> -- Improve error messages when tdc aborts its run
> +- Improve error messages when tdc aborts its run.  Partially done - still
> +  need to better handle problems in pre- and post-suite.
>
> -- Allow tdc to write its results to file
> +- Use python logger module for debug/verbose output
> +
> +- Allow tdc to write its results to file.
> +  Maybe use python logger module for this too.
> +
> +- A better implementation of the "hooks".  Currently, every plugin
> +  will attempt to run a function at every hook point.  Could be
> +  changed so that plugin __init__ methods will register functions to
> +  be run in the various predefined times.  Then if a plugin does not
> +  require action at a specific point, no penalty will be paid for
> +  trying to run a function that will do nothing.
> +
> +- Proper exception handling - make an exception class and use it
> +
> +- a TestCase class, for easier testcase handling, searching, comparison
> +
> +- a TestSuite class
> +  and a way to configure a test suite,
> +  to automate running multiple "test suites" with different requirements
> +
> +- super simple test case example using ls, touch, etc
> --
> 2.15.1
>

Powered by blists - more mailing lists