linux-kernel - Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <6AFCFCC5-DBC2-403E-9EED-D2A85FFCB21C@mit.edu>
Date:	Fri, 4 Mar 2011 13:58:45 -0500
From:	Theodore Tso <tytso@....EDU>
To:	CAI Qian <caiqian@...hat.com>
Cc:	subrata@...ux.vnet.ibm.com, ltp-list@...ts.sf.net,
	vapier@...too.org, linux-kernel@...r.kernel.org,
	akpm@...ux-foundation.org, torvalds@...ux-foundation.org,
	Paolo Ciarrocchi <paolo.ciarrocchi@...il.com>
Subject: Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011.

On Mar 2, 2011, at 8:52 PM, CAI Qian wrote:

> Those days, there just too many tests and testing projects for kernel like
> LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and
> then to extract the best?

Part of the problem is that every single testing project has different goals and
priorities.   For example xfstests is maintained by the XFS folks, as well as people
from some of the other file system development efforts (the ext4 one in particular,
thanks to people like Eric Sandeen), to test file systems.

At least at one point, I had heard a complaint that LTP was more focused on 
increasing test coverage as measured by a code coverage tool in the kernel
than it was about about covering edge conditions, or races.  There's nothing 
wrong with that, per se, and I don't know if it was true then or now, but it's a very
different focus from one which is focused increasing the data reliability of file
systems, quickly and efficiently.

And then there's the LSB test suites, which is really code at testing correctness
from a standards perspective, which is a different focus yet again from the LTP
and xfstests approach.

Bottom line, I'm a big fan of having different test suites, with different philosophies.
Each philosophy has its strengths and blind spots, and so a problem that might
be missed by one test suite might get caught by another.

The only real problem is an operational one.   There are some programs which
are used by both LTP and xfstests, and changes that is made in one, don't 
necessarily get propagated to the other unless someone manually does it.
But I think we can solve that without trying to merge all of these tests into a
single Grand Unified Test Suite.

-- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/