Message-ID: <1350979382-19983-1-git-send-email-steffen.schwigon@amd.com>
Date:	Tue, 23 Oct 2012 10:03:00 +0200
From:	<steffen.schwigon@....com>
To:	<linux-kernel@...r.kernel.org>
CC:	<ss5@...ormalist.net>, <srostedt@...hat.com>,
	<Dariusz.Filipski@...arch.com>,
	<yoshitake.kobayashi@...hiba.co.jp>, <sassmann@...hat.com>,
	<andre.przywara@....com>, <frank.arnold@....com>,
	<conny.seidel@....com>, <csd@...adcom.com>,
	Steffen Schwigon <steffen.schwigon@....com>
Subject: [PATCH 0/2] RFC: A common test container

From: Steffen Schwigon <steffen.schwigon@....com>

OVERVIEW
--------

ABSTRACT: This patch set introduces a mature test library and t/ as a
subdirectory for placing tests that can be run via the existing Linux
tool 'prove' (available in most major Linux distributions). I already
presented and discussed the approach with several people at LinuxCon
Europe 2011 as part of my talk "Linux and virtualization testing".

Tests in t/ are assumed to produce TAP, the "Test Anything Protocol"
(see http://testanything.org) which is well established in many test
communities. The advantage for Linux is that it is easy to generate
which makes it suitable for restricted environments like OS testing.

To write such tests for Linux a library with special utilities is
provided: 'bash-test-utils'.

It provides utility functions to write tests in baremetal and Xen/KVM
contexts (host and guest support) and was polished to work with i686,
x86_64, and ARM under lots of major Linux distributions (Debian,
Ubuntu, RHEL, SLES, Slackware, Fedora, openSUSE). We actively use this
library in all the above contexts for about 2 years.

Other TAP libraries (without special Linux support) are available for
many programming languages.

The autotest project already provides TAP support.


PURPOSE
-------

Provide a container subdir where to place functional tests that are
*in sync* with Linux' git history.

Such tests are usually "unit tests" for features and can also be used
for bisecting regressions of those feature, e.g., by executing 'prove'
with 'ktest' (already part of Linux).

It is possible to organize such tests in a subdir hierarchy (though
not contained with this patch) where separate functional areas as well
as Linux contributors and vendors can be organized without conflicts
and executed separately, for example:

 t/
 t/bare/arch/x86
 t/bare/arch/arm
 t/virt/kvm/
 t/virt/xen/
 t/vendors/fedora/
 t/vendors/companyx/

Separate testprojects like autotest can be kept in sync with the Linux
kernel via git submodules under t/ and called via frontend wrapper
scripts to trigger TAP output.


SYNOPSIS
--------

Testscripts are by convention named "*.t" to be chosen automatically
by 'prove' if executing directories and to differentiate between tests
and other utilities. The subdirectory t/ is also a default convention
used by 'prove'.

Tests can be executed like this:

  $ prove -v t/example/basic.t
  $ prove -v t/example/
  $ prove -r t/


QUESTIONS
---------

Q: Why not submit it to one of the Linux test projects, like LTP or
   autotest?
A: External test projects are not in sync with the Linux kernel. They
   regularly do not fit older or newer kernels.

   Having a test container (subdir t/ with the TAP convention) in
   Linux itself allows

    - contributors, vendors, and companies to place a functional test
      nearby the functionality it belongs to.
    - integrate and sync with existing test projects, e.g., via git
      submodules or wrapper scripts.


Q: Why TAP? Why not XUNIT or other test protocols?
A: There are not that many test protocols. TAP is well established,
   easy, language and platform independent, and already supported in
   every Linux system out-of-the-box. It looks like this:

     1..4
     ok - feature a
     not ok - feature b
     # diagnostic lines with details
     ok - feature c # SKIP not done for this architecture
     not ok - feature d # TODO yet to be implemented

   This can be produced even with the most minimal tools like printf
   from a test inside a strange subsystem, though for practically all
   major programming languages libraries exist. And for evaluating
   tests the tool 'prove' already exists in your Linux system.

   The syntax is easy, yet allows expressing complex test results if
   needed. Besides the #TODO/#SKIP semantics it also allows to embed
   structured data, eg. for benchmark results.


Q: Do we need more framework overhead to execute the tests?
A: Not in the Linux kernel. There are enough tools to execute them:

     - prove ... to run functional tests
     - ktest ... to compile a kernel and boot a machine

   Everything is in place, we just need the common container, per
   convention.


Steffen Schwigon (2):
  bash-test-utils: support writing functional tests
  A common test subdirectory

 t/examples/basic.t                    |   49 ++
 tools/testing/tap/bash-test-utils     | 1508 +++++++++++++++++++++++++++++++++
 tools/testing/tap/bash-test-utils.txt |  320 +++++++
 3 files changed, 1877 insertions(+)
 create mode 100755 t/examples/basic.t
 create mode 100755 tools/testing/tap/bash-test-utils
 create mode 100644 tools/testing/tap/bash-test-utils.txt

-- 
1.7.10.4


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists